World-Consistent Video-to-Video Synthesis

We present a GAN-based approach to generate 2D world renderings that are consistent over time and viewpoints, which was not possible with prior approaches. Our method colors the 3D point cloud of the world as the camera moves through the world, coloring new regions in a manner consistent with the already colored world. It learns to render images based on the 2D projections of the point cloud to the camera in a semantically consistent manner while robustly dealing with incorrect and incomplete point clouds. Our proposed approach further shortens the gap between classical graphics rendering and neural rendering.

Colorization of the world's 3D point cloud
Simultaneously rendered 2D output