DynPoint: Dynamic Neural Point For View Synthesis

Method

Stage 1 — correspondence. A pre-trained flow network and a monocular depth network give optic flow and relative depth; triangulation against the 2K neighbouring frames, masked by correspondence / geometric-edge / dynamic masks, fixes the scale and shift of every depth map (consistent depth). A small MLP then predicts the forward and backward scene flow of every surface point, chained across frames and supervised by re-projection onto the optic flow with a cycle constraint. Stage 2 — aggregation and rendering. The 2K reference frames are lifted to 3D, moved to the target time with the scene flow and merged with the target frame into a hierarchical neural point cloud. A rendering MLP aggregates the neural points around each query point and predicts density and colour, which are volume-rendered into the novel view.

Stage 1: Consistent Depth and Scene Flow (interactive, one frame per video)

The consistent-depth point cloud of one frame, lifted to 3D (drag to rotate, scroll to zoom). Colour by RGB or by depth. Yellow→red segments are the scene flow of the dynamic points to the frame 3 steps later; the blue cloud shows the dynamic points of that later frame at their own position. Blue frustum = the camera of the frame.

Stage 2: View Synthesis on the Nvidia Dynamic Scene Dataset

Each video: three frozen moments of the video (frames 6, 12, 18); for each, the camera sweeps ±2 training cameras along the capture trajectory. Input frame  |  synthesised novel view  |  rendered depth. 540×288. Videos use the visualisation filter (reference points at depth discontinuities pruned, sharper neighbour selection); see note under the table.

Comparison with the paper and baselines (PSNR ↑ / LPIPS ↓)

Numbers from Table 1 of the paper (Nvidia Dynamic Scene dataset, DynamicNeRF 12-frame split; training time per scene in brackets).

In-the-Wild Videos (not in the paper)

DAVIS 2017 sequences with no camera information: poses, intrinsics and the depth prior come from a feed-forward model (Depth-Anything-3), the DAVIS object annotation is the dynamic mask, and the rest of the pipeline is unchanged (dynamic points taken from the nearest references only). 40 frames per sequence; three frozen moments (frames 10, 20, 30), camera sweeping along the trajectory: input | novel view | depth.

BibTeX

@inproceedings{zhou2023dynpoint,
  title     = {DynPoint: Dynamic Neural Point For View Synthesis},
  author    = {Zhou, Kaichen and Zhong, Jia-Xing and Shin, Sangyun and Lu, Kai and Yang, Yiyuan and Markham, Andrew and Trigoni, Niki},
  booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
  year      = {2023}
}

Acknowledgements

Our research is supported by Amazon Web Services in the Oxford-Singapore Human-Machine Collaboration Programme and by the ACE-OPS project (EP/S030832/1). The Nvidia Dynamic Scene videos are from Yoon et al. (CVPR 2020) via the NSFF release; the in-the-wild sequences are from DAVIS 2017. This page uses the Nerfies template.