Extracting Neural Materials from Images

Abstract

Neural materials can represent complex specular reflections and scattering effects in a compact, universal basis. However, acquiring and authoring such materials remains challenging. We present NeuMatEx, a differentiable inverse rendering method for extracting spatially varying neural materials from images. The nonlinear structure of neural material latent spaces makes optimization with naïve inverse rendering infeasible. To address this, we train a Large Material Reconstruction Model (LMRM) that directly predicts initial base color, neural material latents, and aleatoric uncertainty guides from images. This material prior provides a good initialization and better constrains our subsequent optimization using inverse path tracing. The predicted uncertainty further helps by anchoring high-confidence regions more tightly to the LMRM prediction, preventing lighting and complex specular effects from being baked into materials. Experiments on synthetic and real assets show that NeuMatEx extracts complex materials with better visual quality and material decomposition than PBR-based methods.

Why neural materials?

Real-time graphics typically relies on the PBR material model (albedo, roughness, metallicity), which cannot express layered materials such as car paint, coated or dusty surfaces. Neural materials (Zeltner et al., 2024; Bitterli et al., 2026; Yu et al., 2026) close this gap with latent textures and a small MLP that runs in real-time inside a path tracer, but have only been baked from existing material graphs. We ask: can we extract neural materials directly from images?

Neural materials express complex multi-layered material reflectance, including effects such as haze, clearcoat, dust, fuzz, scattering and their combinations, in a compact low-dimensional latent representation, enabling efficient real-time rendering.

Why is this hard?

PBR inverse rendering works because the representation is highly constrained. A neural latent space is not: it has many more degrees of freedom, it is observed only through a nonlinear decoder, and the manifold of valid materials can be fractured. Gradient-based inverse rendering from a random start gets stuck in poor local minima, and many combinations of albedo and specular latent explain the same pixels equally well. Optimizing neural material latents naïvely reaches only 12–14 dB albedo PSNR in our experiments.

NeuMatEx method

NeuMatEx in two stages. (a) Given input images and the mesh, the LMRM predicts a feature triplane in a single forward pass, decoded by two MLPs into an initial neural material and per-material uncertainty. (b) We then refine the triplane with differentiable path tracing against the input images. The uncertainty from (a) allows flexible drift in uncertain regions and anchors confident ones, steering the solution away from local minima.

Stage 1: Large Material Reconstruction Model

We repurpose a pretrained video diffusion transformer as a single-step regressor that maps multi-view images to a neural material triplane. Two small decoders read the triplane at each surface point: a material MLP predicts albedo and the neural specular latent, and an uncertainty MLP predicts how confident that prediction is.

Stage 2: Test-time optimization with uncertainty-guided material regularization

Why test-time optimization?

The LMRM prediction is a good starting point, but fine details are blurred and colors and the material decomposition can drift (right, (a)). We refine it with a differentiable path tracer (NVDiffRecMC) extended to neural materials, optimizing only the triplane against the input images with a rendering loss. This recovers fine details and corrects the errors (right, (b)).

Uncertainty-guided material regularization

The rendering loss alone is ambiguous: the optimizer can bake highlights or lighting into the material. We therefore pull the optimized material toward the LMRM prediction, weighted by its predicted confidence: confident regions stay anchored, while uncertain ones, typically complex specular parts, remain free to fit the rendering loss.

Relightable neural materials from multi-view images

PBR vs. Neural materials

We compare NeuMatEx against PBR material extraction methods on test objects with ground-truth neural materials, relit under held-out HDR probes. PBR cannot represent these complex SVBSDFs and bakes specular effects into the albedo; NeuMatEx reproduces them and recovers a clean albedo. More objects in the results gallery.

Giving PBR baselines the same test-time optimization (LSRM++, NVDiffRecMC++) narrows the gap but does not close it. The bottleneck is the representation: a few PBR parameters cannot express multi-lobe, layered reflectance, so the optimizer has to bake it into the albedo, whereas the neural material latent space can represent it directly.

Material extraction on test meshes with ground-truth neural materials, including relighting under held-out environment maps. Best per group in green.

Neural materials from real-world photos

We apply NeuMatEx to real captures from the Digital Twin Catalog (DTC). The extracted neural materials capture effects beyond standard PBR, such as the clearcoat on the teapot and the mallard, and relight faithfully under novel illumination without baked-in lighting.

Citation

@article{youwang2026neumatex,
  author  = {Kim Youwang and Jon Hasselgren and Peter Kocsis and
             Andrea Weidlich and Tae-Hyun Oh and Jacob Munkberg},
  title   = {{Extracting Neural Materials from Images}},
  journal = {arXiv preprint, 2606.26715},
  year    = {2026}
}

Paper

Extracting Neural Materials from Images
Kim Youwang, Jon Hasselgren, Peter Kocsis, Andrea Weidlich, Tae-Hyun Oh, and Jacob Munkberg

article Preprint (arXiv)

movie Video

format_quote BibTeX