NVIDIA Research
Recent advances in differentiable rendering have enabled high-quality reconstruction of 3D scenes from multi-view images. Most methods rely on simple rendering algorithms: pre-filtered direct lighting or learned representations of irradiance. We show that a more realistic shading model, incorporating ray tracing and Monte Carlo integration, substantially improves decomposition into shape, materials and lighting. Unfortunately, Monte Carlo integration provides estimates with significant noise, even at large sample counts, which makes gradient-based inverse rendering very challenging. To address this, we incorporate multiple importance sampling and denoising in a novel inverse rendering pipeline. This substantially improves convergence and enables gradient-based optimization at low sample counts. We present an efficient method to jointly reconstruct geometry (explicit triangle meshes), materials, and lighting, which substantially improves material and light separation compared to previous work. We argue that denoising can become an integral part of high quality inverse rendering pipelines.
NVDIFFREC successfully reconstructs complex geometry from multi-view images, but struggles with the material and light separation. On the left side, we visualize split-screens of the rendered reconstruction and the diffuse albedo texture. Note that NVDIFFREC bakes most of the lighting in the albedo texture, which hurts quality in relighting scenarios (shown to the right). In contrast, by leveraging a more advanced renderer, we successfully disentangle material and lighting (note the lack of shading in the albedo texture), and improve relighting quality. The dataset consists of 200 views of the Rollercoaster from LDraw resources.
Manipulations of our extracted 3D model of the Family (part of the Tanks and Temples dataset) in Blender. We show scene edits, material edits, and relighting. Tree and bird models in the scene edit example are from TurboSquid. .
We show explicit decomposition of shape, materials and lighting, directly from photos with known poses. Family is part of the Tanks and Temples dataset, Character is part of the BlendedMVS dataset and Gold Cape is part of the NeRD dataset.
We show the benefits of denoising on the Porsche scene from LDraw resources. At low sample counts, denoising helps both with geometric reconstruction (in the cockpit) and to capture specular highlights. Even at 128 samples per pixel, denoising improves specular highlight and high frequency lighting details.
The initial guess for topology are randomized SDF values on a tetrahedral grid. After 1000 iterations, we obtain high quality topology and plausible materials and lighting for this complicated asset. Synthetic dataset with 200 frames, generated from a part of the Apollo capsule, courtesy of the Smithsonian.
@article{hasselgren2022nvdiffrecmc,
author = {Jon Hasselgren and Nikolai Hofmann and Jacob Munkberg},
title = "{Shape, Light, and Material Decomposition from Images using Monte Carlo Rendering and Denoising}",
journal = {arXiv:2206.03380},
year = {2022}
}
Shape, Light, and Material Decomposition from Images using Monte Carlo Rendering and Denoising
Jon Hasselgren, Nikolai Hofmann and Jacob Munkberg