NVIDIA Research
Shape, Light, and Material Decomposition from Images using Monte Carlo Rendering and Denoising

Shape, Light, and Material Decomposition from Images using Monte Carlo Rendering and Denoising

NeurIPS 2022

We learn topology, materials, and environment map lighting jointly from 2D supervision. We directly optimize topology of a triangle mesh, learn materials through volumetric texturing, and leverage Monte Carlo rendering and denoising. Our output representation is a triangle mesh with spatially varying 2D textures and a high dynamic range environment map, which can be used unmodified in standard game engines. Knob model by Yasutoshi Mori, adapted by Morgan McGuire.


Recent advances in differentiable rendering have enabled high-quality reconstruction of 3D scenes from multi-view images. Most methods rely on simple rendering algorithms: pre-filtered direct lighting or learned representations of irradiance. We show that a more realistic shading model, incorporating ray tracing and Monte Carlo integration, substantially improves decomposition into shape, materials and lighting. Unfortunately, Monte Carlo integration provides estimates with significant noise, even at large sample counts, which makes gradient-based inverse rendering very challenging. To address this, we incorporate multiple importance sampling and denoising in a novel inverse rendering pipeline. This substantially improves convergence and enables gradient-based optimization at low sample counts. We present an efficient method to jointly reconstruct geometry (explicit triangle meshes), materials, and lighting, which substantially improves material and light separation compared to previous work. We argue that denoising can become an integral part of high quality inverse rendering pipelines.

Video illustrating our training progress and scene editing examples. All examples enabled by our explicit decomposition into a triangle mesh, PBR materials and HDR environment light, directly compatible with traditional graphics engines. Feel free to download the video, native resolution: 1024x1024 pixels.

Improved material and light separation

NVDIFFREC successfully reconstructs complex geometry from multi-view images, but struggles with the material and light separation. On the left side, we visualize split-screens of the rendered reconstruction and the diffuse albedo texture. Note that NVDIFFREC bakes most of the lighting in the albedo texture, which hurts quality in relighting scenarios (shown to the right). In contrast, by leveraging a more advanced renderer, we successfully disentangle material and lighting (note the lack of shading in the albedo texture), and improve relighting quality. The dataset consists of 200 views of the Rollercoaster from LDraw resources.

Scene manipulation in standard DCC tools

Manipulations of our extracted 3D model of the Family (part of the Tanks and Temples dataset) in Blender. We show scene edits, material edits, and relighting. Tree and bird models in the scene edit example are from TurboSquid. .

Explicit decomposition of shape, materials, and lighting

We show explicit decomposition of shape, materials and lighting, directly from photos with known poses. Family is part of the Tanks and Temples dataset, Character is part of the BlendedMVS dataset and Gold Cape is part of the NeRD dataset.

Benefits of denoising

We show the benefits of denoising on the Porsche scene from LDraw resources. At low sample counts, denoising helps both with geometric reconstruction (in the cockpit) and to capture specular highlights. Even at 128 samples per pixel, denoising improves specular highlight and high frequency lighting details.

Visualization of the optimization process

The initial guess for topology are randomized SDF values on a tetrahedral grid. After 1000 iterations, we obtain high quality topology and plausible materials and lighting for this complicated asset. Synthetic dataset with 200 frames, generated from a part of the Apollo capsule, courtesy of the Smithsonian.


    author = {Jon Hasselgren and Nikolai Hofmann and Jacob Munkberg},
    title = "{Shape, Light, and Material Decomposition from Images using Monte Carlo Rendering and Denoising}",
    journal = {arXiv:2206.03380},
    year = {2022}


Shape, Light, and Material Decomposition from Images using Monte Carlo Rendering and Denoising

Jon Hasselgren, Nikolai Hofmann and Jacob Munkberg

description arXiv version
description Video
insert_comment BibTeX