NVIDIA Real-Time Graphics Research
NVIDIA Research
Neural materials can represent complex specular reflections and scattering effects in a compact, universal basis. However, acquiring and authoring such materials remains challenging. We present NeuMatEx, a differentiable inverse rendering method for extracting spatially varying neural materials from images. The nonlinear structure of neural material latent spaces makes optimization with naïve inverse rendering infeasible. To address this, we train a Large Material Reconstruction Model (LMRM) that directly predicts initial base color, neural material latents, and aleatoric uncertainty guides from images. This material prior provides a good initialization and better constrains our subsequent optimization using inverse path tracing. The predicted uncertainty further helps by anchoring high-confidence regions more tightly to the LMRM prediction, preventing lighting and complex specular effects from being baked into materials. Experiments on synthetic and real assets show that NeuMatEx extracts complex materials with better visual quality and material decomposition than PBR-based methods.
Given a 3D model and a set of multi-view images, we extract high quality neural materials using our differentiable inverse rendering method. We show the extracted albedo, neural specular latents, and relit renderings with four different HDR probes.
NeuMatEx faithfully reconstructs a wide variety of neural materials from images. Reconstructed neural materials demonstrate complex multi-lobe specular effects including haze, clearcoat, dust, fuzz, scattering and even their combinations.
Hunyuan3D-2.1 and TRELLIS.2 are monocular feed-forward PBR estimation models included to highlight the limitations of PBR, and do not represent a fair comparison to optimization methods. NVDiffRecMC++ is our own extension of NVDiffRecMC with a feed-forward PBR initialization, serving as a strong optimization-based PBR extraction method. PBR-based methods fail to represent complex SVBSDFs, instead baking specular components into the base color (see insets). NeuMatEx faithfully decomposes these materials by optimizing in the neural material latent space, with a clean base color free of baking and specular artifacts.
We apply NeuMatEx to real-world photographs from the DTC dataset. Despite the DTC dataset being composed mostly of simple materials and not designed with neural material reconstruction in mind, the extracted neural materials capture effects beyond the standard PBR, e.g., clearcoat on the teapot and the mallard.
@article{youwang2026NeuMatEx,
author = {Kim Youwang and Jon Hasselgren and Peter Kocsis and Andrea Weidlich and Tae-Hyun Oh and Jacob Munkberg},
title = {{NeuMatEx: Extracting Neural Materials from Multi-view Images}},
journal = {arXiv preprint, 2606.26715},
year = {2026}
}
NeuMatEx: Extracting Neural Materials from Multi-view Images
Kim Youwang, Jon Hasselgren, Peter Kocsis, Andrea Weidlich, Tae-Hyun Oh, and Jacob Munkberg