Texture Space Material Diffusion

Abstract

We present a method for generating high quality materials for 3D objects entirely in texture space. We finetune a video diffusion transformer for text-guided material generation, multi-view material generation, and material upscaling. Our key insight is to use the known projection from image space to texture space, enabling the diffusion process to generalize across arbitrary geometries and texture parameterizations. This approach also avoids the view consistency issues inherent in video and multi-view diffusion models. Because texture space is two dimensional, we can reuse the strong priors of pretrained video diffusion models. We apply our method to high quality material reconstruction from posed photos captured under unknown lighting, as well as to text- and image guided material generation. Our method can scale to high resolutions (8K), 100+ input views, and neural material representations. In quantitative and qualitative evaluations we show state-of-the-art results for material generation and reconstruction.

Why texture space?

Image-space methods generate multiple views and reconstruct materials from them, but the same surface point may be predicted differently across views, leading to blur, lost detail and suppressed highlights. Object-space methods (sparse voxels, triplanes) are view consistent but memory hungry and trained from scratch. Texture space offers the best of both: it is view consistent by design and two-dimensional, so powerful pretrained image and video diffusion priors can be reused directly. TEXGen (Yu et al., 2024) showed that it is possible to directly generate albedo maps in texture space. Inspired by this, we formulate a texture space generation process for complete PBR materials.

Texture space observations

Geometry, parameterization, multi-view captures, and texture space observations. Given camera poses and geometry with a non-overlapping UV parameterization, each view is projected back into texture space, producing partially covered observations. Formulating diffusion in this 2D domain lets us leverage powerful image and video diffusion priors.

Method

Method overview. A finetuned DiT video model operates on texture space inputs. In all modes the model is conditioned on texture space world positions, normals and a text prompt. It demodulates the unknown lighting and produces clean albedo, roughness and metallicity maps with high frequency detail, ready for standard content creation tools.

We finetune Wan 2.1-1.3B with a flow matching objective. The texture space views, the shared conditions (world space positions and normals), and the noisy material latents are concatenated along the temporal dimension, so the transformer can attend across all of them. The output is two texture space images: base color, and an RGB-packed height / roughness / metallicity map. Training uses 120k path-traced objects from TexVerse, 17 frames per object, with resolution progressively increased from 512² to 2048².

Inference scaling: 8K textures and 100+ views

Progressive noise rolling. We generate at 2K, upscale to 8K, add a small amount of noise and resume denoising on non-overlapping 2K crops, randomly rolling the crop grid each step to hide seams. Coverage-aware expert aggregation. Input views are split into overlapping batches; each batch predicts the flow for the same noisy latent. Per texel, we keep the top-2 experts by view coverage and average them weighted by coverage, keeping redundancy without over-smoothing.

17 views at 2K vs. 117 views at 8K. Our inference augmentations capture high-frequency detail and more accurate specular predictions while preserving fidelity (28.42 dB / 0.919 / 0.0668 → 28.34 dB / 0.917 / 0.0639 PSNR / SSIM / LPIPS). Asset from the Digital Twin Catalog dataset.

Material reconstruction from real photos

Reconstruction from posed photographs of two objects from the DTC dataset, shown under the capture lighting and two novel lighting conditions. DiffPT (differentiable path tracing) bakes lighting into the materials, while LSRM jointly reconstructs geometry and materials. Our method yields clean, relightable materials. Assets from the Digital Twin Catalog dataset.

Multi-view to materials from 17 views at 2K. DiffPT overfits the capture lighting (best real reconstruction) but fails to generalize under novel lighting, where our method leads by over 5 dB.

Generative materials

Single image → material

Materials conditioned on a single image and text description, compared with Hunyuan3D 2.1, VideoMatGen and Trellis.2 using the same reference geometry and conditioning view. The visualized view is rotated slightly from the conditioning view (leftmost insets).

All metrics evaluated on 32 scenes from from the BlenderVault test set, using 8 views × 8 light probes for each scene.

Text → material

Materials generated from a text prompt and the input geometry only (UV mask, world space positions and normals), compared with VideoMat and VideoMatGen. Without any image guidance, the 3D-aware RoPE gives enough structure to produce semantically meaningful materials.

Material components

Rendered-image metrics measure the final appearance, but not whether each material parameter is recovered correctly. We therefore evaluate the individual components — base color, roughness and metallicity — of materials reconstructed from 17 views. Because LSRM reconstructs its own geometry, and hence a different UV space, the comparison is made on g-buffer renderings of the material maps rather than on the texture images themselves: 17 views of each of the 32 scenes in the BlenderVault test set.

Figure S2: Material components. G-buffer renderings of base color (adjusted for intensity), roughness and metallicity for DiffPT, LSRM, our method and the reference. Our maps stay closest to the reference on all three components, with a clean base color and roughness that follows the surface detail.

Reconstructed base color carries a systematic intensity shift, since observed brightness depends jointly on illumination and reflectance. To compensate, we additionally report scale-invariant PSNR.

Table S3: Material components. Metrics on g-buffer renderings, 32 scenes × 17 views, from multi-view guidance. siPSNR is the scale-invariant PSNR of the base color; the remaining columns are standard PSNR.

Limitations

Inference is not yet optimized: the 17-view model takes about 130 s at 2K on a GB300 GPU. Capturing specular appearance remains challenging. Wan 2.1 attends over 16×16 patches, so patches spanning several UV atlas segments cannot be fully disambiguated; patch-aware unwrapping or pixel-space diffusion are promising directions.

Citation

@article{munkberg2026texdiff,
    author  = {Jacob Munkberg and Peter Kocsis and Jon Hasselgren},
    title   = {{Texture Space Material Diffusion}},
    journal = {arXiv preprint arXiv:2609.37654},
    year    = {2026}
}

Paper

Texture Space Material Diffusion
Jacob Munkberg, Peter Kocsis, and Jon Hasselgren

article Preprint (arXiv)

format_quote BibTeX