Neural appearance models in Three.js

Load a normal-mapped, clearcoated surface. A car hood, a brushed metal panel, anything with fine structure and a specular response. Walk the camera back.

Around mid-distance the highlights start to shimmer. Keep going and they stop shimmering, which is worse: they dissolve into a flat grey wash that looks nothing like the material up close. The surface has lost its character and picked up an aliasing artifact on the way out.

You know why this happens. You have probably fixed it three different ways in three different engines, and none of the fixes generalised. The reason is more specific than "filtering is hard," and that version of the problem points at a material format where a texture texel and an MLP's weights are the same kind of thing, both trained rather than authored. Format, training data, hand-written backpropagation on a platform with no autodiff, the Three.js forward pass, and where the approach still falls short.

The problem: mipmaps lie#

The case where mipmapping works#

Albedo mips work. That reliability makes it easy to skip the justification.

A pixel at distance covers some patch of the surface. The colour should be the average of the patch. Diffuse reflectance combines linearly: half of one albedo plus half of another produces the appearance of a surface that is half each. The average of the texels is the correct answer. Box-filter your way down the pyramid and you get the right result at every level.

Linearity is what makes the average correct. That is a special property of albedo, not of material parameters in general.

The case where it doesn't#

Take a normal map. Two adjacent texels, one tilted left, one tilted right. Average them and you get a normal pointing straight up.

A surface that is half left-facing and half right-facing does not look like a flat surface. It looks like a surface with two highlights. You have replaced a bimodal distribution with its mean, and the mean describes neither of the things it came from. At distance, every V-groove in your normal map becomes a flat plane.

Roughness is the same failure in another channel. Roughness parameterises the width of a specular lobe. It is already a summary statistic about microscopic variation. Average two roughness values and you get a lobe that is neither the sharp one nor the broad one, and not the sum of the two either. The variance you destroyed when you averaged the normals was supposed to reappear as extra roughness. It doesn't, because nothing in the pipeline connects those two channels.

The graphics literature has a lineage of partial fixes.

Toksvig measures the length of the averaged normal. Short means the normals disagreed. It converts that shortfall into extra roughness. It patches the leak above.

LEAN mapping stores second moments of the normal distribution alongside the first, so the filter has enough information to reconstruct an anisotropic lobe rather than an isotropic one. LEADR extends it to displacement.

Roughness clamping does not fix the average. It refuses to let roughness go below a floor that depends on distance, trading correctness for stability.

Each of these is careful, well-founded work, and each covers one term of one BSDF. None of them tells you what to do with a clearcoat layer whose normal map disagrees with the base layer's, or with a sheen term, or with a thin-film interference model, or with whatever your artists wired up in MaterialX last Tuesday.

The correct average of two materials is not the average of their parameters.

Filtering operates on the storage format. You designed that format for a human. You decided channel 0 would be roughness because you need to reason about roughness. Nothing in that decision was optimised for the channel averaging correctly. Albedo happens to. Everything else has been riding that accident.

The other cost#

A second problem sits next to the first.

Open a production MaterialX graph. Base layer, clearcoat, sheen, maybe a fuzz term and a thin-film. Each layer has its own normal, its own roughness, its own Fresnel response, and the graph that computes them is hundreds of nodes deep before you reach the BSDF that consumes the result.

All of that evaluates per fragment, per light. The cost scales with how complicated your artists were feeling, which is a difficult thing to put in a performance budget. And it scales again with light count, because every layer's Fresnel and every layer's lobe has to be re-evaluated for each incoming direction.

The standard answer is to bake: collapse the graph into a fixed set of textures and evaluate a single fixed BSDF. That works, and it is what studios do, and it reintroduces problem number one. You have baked into a format whose channels have human-assigned meanings, and now the mip chain lies to you again.

The two problems have the same shape#

You want two things:

1. A representation whose average is meaningful, so the mip chain tells the truth.
2. An evaluator whose cost is fixed, so the budget survives contact with artists.

Both ask you to stop choosing the storage format by hand.

Roughness averages badly because it is roughness: it carries a specific, human-legible meaning that constrains how it can be combined. The graph is expensive because it is a graph: an artist-authored structure whose shape you do not control.

Keep the texture. Drop the requirement that channels mean anything in particular. Store some numbers per texel. Let training decide what they encode. The encoding only has to be learnable, so you can optimise it for those two properties instead of for human readability.

That is the idea.

Nothing about it requires neural networks in principle. In practice a small neural network is the most convenient tool for learning an encoding and its decoder at the same time. The next section covers the vocabulary in NeuralAppearanceMLP.js, and no more.

An MLP is a shader: a crash course#

There is a great deal of machine learning. Almost none of it is relevant here.

This section covers multi-layer perceptrons, ReLU, gradient descent, backpropagation, and Adam, the only vocabulary the rest of this piece needs. It does not cover convolutions, attention, transformers, batch normalisation, dropout, or language models, because the system described here does not use them. If a concept does not appear in NeuralAppearanceMLP.js, it does not appear here.

You have already written the part you need.

An MLP is a shader#

The entire architecture:

vec3 evaluate(float x[20]) {
    float h1[32];
    for (int j = 0; j < 32; j++)
        h1[j] = max(0.0, dot(W0[j], x)  + b0[j]);

    float h2[32];
    for (int j = 0; j < 32; j++)
        h2[j] = max(0.0, dot(W1[j], h1) + b1[j]);

    vec3 out;
    for (int j = 0; j < 3; j++)
        out[j] = dot(W2[j], h2) + b2[j];

    return out;
}

A dot product, an add, a clamp at zero. Three times. That is the neural network this system uses, at its actual size.

Vocabulary, applied to code you have already read:

• Each for loop is a layer.
• W are the weights, b the biases. Together, the parameters.
• max(0.0, ·) is the activation function, specifically ReLU (rectified linear unit).
• h1 and h2 are hidden layers. Hidden means "not the input and not the output." They are intermediate registers.
• 32 is the hidden size. You choose it before training and cannot change it afterwards, because it fixes the shape of W0, W1, W2.
• Running this function is a forward pass.

Two hidden layers. Around 1,800 parameters. Small enough that you could write the weights out on a few sheets of paper.

Why the clamp is not optional#

max(0.0, x) looks like an afterthought.

Take it out. Without it, layer two computes W1 · (W0 · x), which is (W1 · W0) · x, which is a single matrix. Stack fifty layers and you still have a single matrix. All that structure collapses into one linear transformation, and a linear function of (z, wi, wo) cannot represent a specular highlight. It cannot represent anything with a peak in it.

The nonlinearity is the only thing standing between you and an expensive matrix multiply. ReLU is the cheapest nonlinearity that works, which is why it is the default.

Each ReLU unit is a hinge. It contributes nothing until its input crosses zero, then contributes linearly. A layer of 32 hinges is a piecewise-linear function with up to 32 folds, and two such layers compose into something that can bend into a highlight shape. More units, more folds, finer approximation.

Training is fitting, and you have done it before#

You have fit a GGX lobe to measured BRDF data. You have projected an environment map into spherical harmonics. You have least-squares fit a polynomial to an expensive function so it could run in a shader.

Training is that. Same activity: you have a reference you trust and an approximation with free parameters, and you adjust the parameters to minimise the error.

The difference is that there is no closed form. With SH projection you compute the coefficients directly by integration. With an MLP there is no integral to evaluate, so you nudge, iteratively, in the direction that reduces the error. That direction is the gradient, and following it downhill is gradient descent.

Concretely, per step:

1. Run the forward pass on a batch of inputs. (A batch is some number of inputs processed together; this system uses 1,024.)
2. Compare against the reference. That comparison is the loss, one number saying how wrong you were.
3. Compute how the loss would change if you nudged each parameter. That is the gradient.
4. Move each parameter a small step against its gradient. The step size is the learning rate.
5. Repeat.

One repetition is an iteration or step. An epoch means one pass over a fixed dataset. It will not come up again here. The training data is generated on demand and never repeats.

Backpropagation, by hand, once#

Step 3 is the chain rule, applied from the output backwards, on a network small enough to hold in your head.

Take a two-layer network with scalar weights:

z1 = w0 · x        a1 = max(0, z1)
z2 = w1 · a1       L  = |z2 − y|

We want ∂L/∂w0 and ∂L/∂w1. Work right to left.

∂L/∂z2  = sign(z2 − y)                        call this δ2
∂L/∂w1  = δ2 · a1                             weight grad: δ out × activation in
∂L/∂a1  = δ2 · w1                             push the error back through the weight
∂L/∂z1  = (z1 > 0) ? ∂L/∂a1 : 0               ReLU's derivative is a mask
∂L/∂w0  = ∂L/∂z1 · x

That is backpropagation. Every layer does the same three things: receive an error signal from above, produce a weight gradient by multiplying it against the activation that came in from below, and pass a modified error signal further down.

Two details from those five lines matter later.

The ReLU derivative is a mask. Where the unit was clamped, gradient stops dead. A unit whose output is always negative can get stuck: no gradient reaches it, so it never moves. The code has a small mitigation for this, covered in the backpropagation section below.

The backward pass consumes the forward pass's activations. Notice a1 appearing in the formula for ∂L/∂w1, and z1 in the ReLU mask. You cannot run backward without having kept what forward produced.

That second point has a direct consequence in the implementation: memory cost scales with batch size, not with parameter count. It is why NeuralAppearanceGPUModel allocates an activationsStorage buffer sized by samples-in-flight, and why the backpropagation section below has a part on recomputing some activations instead of storing them.

Adam, briefly#

Plain gradient descent takes the same size step for every parameter. That fails when some parameters have consistent large gradients and others have tiny ones.

Adam keeps two running averages per parameter: the mean of recent gradients (momentum, so it keeps moving through flat regions) and the mean of recent squared gradients (so it can normalise the step size per parameter). It divides one by the square root of the other, giving each parameter its own effective learning rate.

That buys robustness to badly-scaled parameters, without hand-tuning a learning rate per tensor. The derivation is available anywhere and you do not need it. What you do need is that Adam requires two extra buffers the same size as your parameters, which, when the parameters include a multi-megabyte latent texture, is a real memory cost. That will come up.

An MLP is three dot-product loops with clamps. Training is fitting. Backprop is the chain rule run backwards, using stored activations. Adam is gradient descent with per-parameter step sizes. Next: using it to replace a texture set.

The latent texture: an auto-decoder, not an encoder#

Keep the texture. Give up on the channels meaning anything. The catch, if you arrive with a little machine learning background, is the auto-decoder.

Your texture set is a hand-designed codec#

A PBR material, structurally:

You have an encoding: base colour in three channels, roughness in one, metallic in one, a tangent-space normal in two or three, maybe occlusion and clearcoat and their own normal and roughness. And you have a decoder: your BSDF, which consumes those channels and produces reflectance.

Both halves were designed by hand. Someone decided that roughness deserved a channel and that it should be perceptually remapped before storage. Someone decided the normal would be tangent-space and two-channel-reconstructed. These are good decisions, made by thoughtful people, optimised for a specific goal: that a human can look at the channel and reason about it.

They were not optimised so the channels would average correctly. That is why they don't.

Give up the meanings, keep the texture#

Store eight numbers per texel. Do not assign them meanings. Learn a small MLP that turns those eight numbers, plus the view and light directions, into reflectance.

You choose the budget: eight channels, a 32-wide MLP. The training process chooses the encoding.

The format is small enough to quote in full. This is NeuralAppearanceFormat.js, the entire contract:

const LATENT_CHANNELS      = 8;
const LATENT_TEXTURES      = 2;
const CHANNELS_PER_TEXTURE = 4;
const DECODER_INPUT_SIZE   = 20;
const ROTATION_OUTPUT_SIZE = 12;

Eight channels split across two RGBA textures, because a texture fetch gives you four components at a time. Half-float storage, so 16 bytes per texel at base resolution. A full power-of-two mip chain down to 1×1. The 20 and the 12 belong to the frame construction covered in the next section.

There is no normal map in that list. No roughness map. No metallic, no clearcoat, no sheen. Eight anonymous channels, and whatever the network decided to put in them.

Where the latents come from#

If you have encountered autoencoders, you know the shape: an encoder network compresses input x into a small code z, a decoder reconstructs x from z, and you train the pair end to end. The natural assumption is that an encoder here looks at your MaterialX textures and emits eight channels.

There is no encoder. Nothing computes the latents from anything.

Each texel's eight floats are free parameters. They are initialised randomly and updated by Adam, on the same footing as the MLP weights. In NeuralAppearanceGPUCompute.js the latent grid and the weight buffer get their own Adam passes, structurally identical:

createAdamWeightsComputeNode( gpuModel )   // network parameters
createAdamLatentsComputeNode( gpuModel )   // the texture

Same optimiser, same momentum buffers, same clipping. The texture is a parameter tensor. It is trained, not computed.

This pattern has a name, an auto-decoder, from the DeepSDF paper in 2019. The three cases get conflated:

Pattern	Where z comes from
Autoencoder	An encoder network computes it from the input
GAN	Sampled from an imposed prior, usually a Gaussian
Auto-decoder	Stored directly and optimised by gradient descent

The closest familiar thing is an embedding table in a language model: a big matrix of free parameters, indexed by token ID, filled in during training. Here you index by UV instead of token ID, and bilinear interpolation makes the lookup differentiable so gradients can flow back into it. That flow is the most elegant thing in the codebase, and the backpropagation section below is largely about it.

One practical consequence, since it will save you a debugging session: because the latents are trained rather than derived, you cannot compute the latent for a new texel without retraining. There is no encoder to run. Editing a neural material is not like editing a texture.

Almost all of it is texture#

Count the parameters. At 512² base resolution with the default hidden size of 32:

Component	Parameters
Latent pyramid, 8 channels, ×4/3 for the mip chain	~2,800,000
Frame layer (8×12 + 12)	108
BRDF MLP (20→32→32→3)	~1,800
Emission head (8→3 linear)	27
Opacity head (8→1 linear)	9
Everything that is not the texture	~1,950

Ninety-nine point nine percent of the learned parameters are the texture.

That ratio reframes the whole system. The MLP is not "the model" in any meaningful sense. It is a shared codec, a small fixed decompressor. The material lives in the latent grid, as it lived in your texture set before. You have replaced a hand-designed encoding with a learned one and kept everything else about how textures work.

If the MLP is a shared codec, it could be shared in fact. Train one decoder across a corpus of materials, ship those two kilobytes of weights once, and vary only the latent grid per asset. NVIDIA's version of this system does something adjacent. This one does not, yet, but the architecture already permits it. The limitations section returns to this.

The compression dial is latentDownsample in the trainer options. The latent grid can be lower resolution than the source textures it was distilled from. You are not obliged to match the input's texel density, because the encoding is denser per texel than roughness-and-a-normal was.

Why the mip chain stops lying#

The mip levels are trained, not filtered.

Nobody box-filters the latent pyramid. Every level is a separate set of free parameters, and every level is fit, during the same training run, from the same optimiser, against a reference that has been correctly prefiltered at that level's footprint. The training-data section below explains how that reference gets produced: Gaussian supersampling the real material over the area a texel covers. The structural point is available now:

Level 3 does not contain the average of level 2's latents. Level 3 contains whatever eight numbers cause the decoder to produce the appearance of a surface patch four times larger. If that means encoding "two competing highlights" rather than "one averaged normal," the network is free to do so, because nothing constrains channel 4 to be a normal component.

The correct average of two materials is not the average of their parameters. So don't average the parameters. Train a parameterisation where the right answer is representable, then fit the right answer directly.

What the channels look like#

Visualise them. There is an obvious debug view where each latent channel is displayed as false colour.

You will also want to interpret them, and you will mostly fail. Occasionally one channel resembles something recognisable, and that is a coincidence of initialisation rather than a design: the network has no incentive to disentangle its encoding into human-legible factors, and every incentive to use all eight channels jointly. Same situation as any learned representation. The encoding works. It is not designed for you to read.

Eight numbers per texel is not enough for a highlight, though. A highlight depends on where you are standing, and so far this hasn't addressed directions at all.

Directions, and the frames the network invents#

A latent code on its own cannot produce a highlight. Reflectance is a function of two directions, where the light is and where you are, and so far the system has been told about neither.

Feed the directions to the MLP alongside the latent. That is correct. The open question is the coordinate frame, and the answer is neither of the two you would reach for first.

World space#

Feed wi and wo in world space and the network has to learn, separately at every texel, which way that texel is facing. Every latent code would need to encode surface orientation just so the decoder could undo it, and the decoder would need enough capacity to handle arbitrary orientations of the same material.

You would be spending a 1,800-parameter network on rediscovering the mesh. That is not viable at this size.

Tangent space#

The standard answer in graphics is tangent space: build a TBN from the mesh's vertex normal, tangent and bitangent, transform the directions into it, and now the material is orientation-independent.

This is right as far as it goes, and it is not quite enough. Tangent space is fixed by the geometry. All the sub-geometric detail that a normal map would normally supply, the fine tilts that make a surface look like brushed metal or leather rather than a plane, would have to be encoded in the latent and re-applied by the decoder, which puts us back in world-space territory, at a smaller scale.

The frame the network chooses#

The network produces the frame.

A single linear layer maps the eight latents to twelve numbers, which are read as two vectors, and those become an orthonormal basis:

n = normalize(W_n · z + [0, 0, 1])
t = normalize(W_t · z + [1, 0, 0])
b = normalize(n × t)

Twelve outputs, ROTATION_OUTPUT_SIZE in the format constants: six per frame, and there are two frames. More on the second one shortly.

The bias terms are the identity. +[0,0,1] for the normal, +[1,0,0] for the tangent. At initialisation, when the weights are small random values, the frame is the standard tangent-space basis. The network learns deviations from the identity frame, which is the right thing to start from and a small piece of initialisation hygiene that would be easy to get wrong.

t is not orthogonal to n, and that is fine. The network emits two independent vectors; the cross product manufactures a genuine orthogonal third. The frame is orthonormal by construction rather than by constraint, which means the optimiser never has to be told to keep it orthonormal. The construction handles it.

This is where your normal map went. A per-texel rotation the network is free to choose, learned from data, is what a normal map is. It is why there is no normal texture in the format and no normal channel among the eight latents. The mesostructure lives in the frame layer, distributed across a hundred and eight weights and the latent codes that drive them.

Two frames#

The system builds two of these, and the reason is the same reason your BSDF has layers.

A clearcoat highlight does not point where the base highlight points. That is what makes it read as a separate coat of material rather than a brighter version of the same surface. Anisotropic reflection needs a tangent direction aligned with the brush or weave, which is unrelated to whatever the base layer's shading normal is doing.

One frame cannot express both. Two can, and two is the cheapest thing that can.

This is the substitution that lets one MLP pass replace a layered BSDF evaluation. You are not evaluating a base lobe plus a clearcoat lobe plus a sheen term and compositing them. You are handing the decoder the directions as seen from two different learned orientations and letting it produce the composite directly.

Twenty numbers#

The decoder input assembles:

per frame k:  (wi·t_k, wi·b_k, wi·n_k, wo·t_k, wo·b_k, wo·n_k)   =  6
two frames                                                       = 12
the raw latent code                                              =  8
                                                                   --
DECODER_INPUT_SIZE                                                 20

Both directions, projected into both frames, plus the latents passed through unchanged. The latents appear twice in effect, once directly, once via the frames they generated, which gives the decoder both the material identity and the material's own idea of which way is up.

In NeuralAppearanceGPUCompute.js the projection is six stores per frame, immediately after the frame is built:

activationsStorage.element( projBase.add( 0 ) ).assign( wi.dot( t ) );
activationsStorage.element( projBase.add( 1 ) ).assign( wi.dot( b ) );
activationsStorage.element( projBase.add( 2 ) ).assign( wi.dot( n ) );
activationsStorage.element( projBase.add( 3 ) ).assign( wo.dot( t ) );
activationsStorage.element( projBase.add( 4 ) ).assign( wo.dot( b ) );
activationsStorage.element( projBase.add( 5 ) ).assign( wo.dot( n ) );

Written into the activation buffer contiguously after the eight latents, so layer 0 reads a flat run of twenty floats. Remember the layout. The backpropagation section walks this same code backwards, and the gradient has to find its way from element 8 of that buffer back into the frame weights and then into the latent grid.

The image that proves it#

If you build one debug view for this system, build this one: render both learned frames as arrows across the surface.

On a brushed metal asset, one frame's tangent should align with the brush direction. Nobody told it to. There is no loss term for "align with anisotropy," no supervision on the frames at all. They are an intermediate quantity, and the only signal reaching them is reflectance error backpropagated through the decoder.

If that alignment appears, the network found real structure in the material rather than memorising a lookup table. It is the most convincing single image this system can produce, and it is worth training an asset specifically to get it.

A trained texture, a learned frame, a small decoder. Nothing has trained it yet.

Where the training data comes from#

There is no pile of labelled examples on disk. There could not be.

What you have is the thing you are trying to replace: a MaterialX graph that correctly computes reflectance and is too slow and filters badly. It is a function you can call. So you call it, a few million times, and the answers become the training set.

This is distillation: a large, slow teacher supervises a small, fast student. The teacher is MeshPhysicalNodeMaterial with your MaterialX graph bound to it. The student is the latent grid plus the decoders.

It is a bake, with one extra dimension#

You have baked lightmaps. Evaluate an expensive function across a surface, store the results, look them up cheaply at runtime. Same structure here.

Except the answer varies with two directions as well as position. A lightmap texel holds one radiance value; a reflectance "texel" would need to hold a whole four-dimensional function of incoming and outgoing direction. You cannot store that, which is why you need something that compresses it, and why the bake and the compressor have to be trained together rather than run in sequence.

One useful consequence of generating data on demand: there are no epochs. An epoch is one pass over a fixed dataset, and there is no fixed dataset. Every batch is freshly sampled, never seen before, never seen again. The trainer's 2,000 iterations are 2,000 distinct batches of 1,024 samples.

Overfitting is not a concern. You cannot memorise a training set you never see twice. If the student is wrong it is because it lacks capacity or the optimisation stalled, not because it memorised.

The interrogation rig#

Getting a MaterialX graph to answer arbitrary questions requires a certain amount of violence.

NeuralAppearanceTeacherEvaluator renders the material into an offscreen half-float atlas, using TSL context overrides to replace the geometric inputs the shader graph would normally read, uv, normalView, tangentView, bitangentView, positionViewDirection, with values pulled from data textures. Each texel of the atlas is one training query, and the overrides mean each query can specify its own surface point and its own view direction, regardless of what geometry is actually being drawn.

The light side is handled by NeuralTeacherLightingModel, which injects a single incoming direction wi with unit white radiance and suppresses indirect lighting. What comes back is the material's isolated response to one direction from one direction, which is what the student needs to learn.

It is a shader-graph interrogation harness. You are driving the real material's inputs and reading its outputs, in bulk, on the GPU.

Where to point the directions#

You now have to choose which (wi, wo) pairs to ask about. This is importance sampling.

Sample directions uniformly over the hemisphere and a sharp specular lobe occupies a tiny solid angle. Almost no samples land in it. The student sees a material that is diffuse, learns a material that is diffuse, and your metal comes out looking like clay.

So NeuralAppearanceSampler draws from a mixture:

Distribution	Share	What it covers
Uniform hemisphere	35%	Broad coverage, no bias
Cosine-weighted	20%	Where diffuse energy goes
Uniform sphere	10%	Below-horizon cases, boundary behaviour
Specular mirror pairs	15%	Exact reflections, the peak
Rusinkiewicz half-angle	20%	Around the peak, at microfacet-relevant angles

The Rusinkiewicz parameterisation reformulates the BRDF domain in terms of the half-vector, the coordinate system microfacet models live in. Sampling power-weighted in half-angle space concentrates queries where a GGX lobe has structure, at whatever roughness the material happens to be.

Same principle as importance-sampling a path tracer, applied one level up: you are importance-sampling the training set.

Normalising out the cosine#

The teacher returns outgoing radiance L_o, which includes the geometric cosine factor. The student should learn the BRDF itself, so the trainer divides it out:

f_r = L_o / max(wi·n, 1e-4)

with minimumTrainingCosine (default 0.05) rejecting samples too close to grazing, where the division amplifies noise without bound.

A note if you are reading the original design document: §2.1 there describes the network as outputting a "cosine-factored BRDF." It does not. Cosine is divided out in training and multiplied back at runtime, which is the opposite arrangement. The document is wrong on this point and has been corrected.

Samples are then weighted, and the weighting is an aesthetic decision as much as a numerical one. highlightLossScale (default 2) over-weights bright specular samples, on the reasoning that a wrong highlight is far more visible than slightly wrong diffuse. You are telling the optimiser where you want its attention.

HDR, non-negotiably#

NeuralAppearanceTeacherReadback reads the atlas back as half-float and refuses to fall back to LDR.

An 8-bit readback clips a 400-nit specular highlight to 1.0. The student then learns, from thousands of consistent examples, that the highlight's correct value is 1.0. It fits that. You get a trained material whose highlights are uniformly flat white with no falloff, and the failure is silent. Loss converges, because the student is reproducing what it was told.

Training data corruption is silent. You get a model correctly fit to the wrong thing.

Prefiltering, which is the whole point#

The latent-texture section claimed the mip pyramid stops lying because the levels are trained rather than filtered. Here is how.

For a coarser mip level, one latent texel covers a larger patch of the source material. So the teacher does not evaluate a point. It evaluates the area. The footprint is computed as det(∇UV) · res², and the tile is supersampled with a Gaussian-weighted kernel of between 1 and 64 taps, sized to match the target level's pixel coverage. mipSamplingDecay (0.9) controls how the sample budget falls off across levels.

Mip level 3's training target is the correct appearance of a patch four times larger, integrated, not approximated by averaging parameters. Level 3's latents are then optimised to reproduce that.

A correct reference at every scale, and a representation flexible enough to fit it.

Every level trains at once. The trainer allocates batchSize × mipLevelCount sample capacity and the whole pyramid receives gradients in every iteration. There is no coarse-to-fine schedule.

Architecture and a source of training data. What remains is the part that moves the numbers, and on the web platform that is the hard part.

Training: backpropagation in a shading language#

Every serious implementation of neural appearance models is written in a language that computes derivatives for you.

NVIDIA built that language on purpose. SLANG.D, presented at SIGGRAPH Asia 2023, adds first-class automatic differentiation to a shading language. You write the forward pass and the compiler generates both forward and reverse gradient propagation, with control over how gradients are stored and accumulated. Their stated motivation is this workload: for tiny neural networks inside a renderer, the overhead of a general ML framework dominates, because frameworks serialise intermediates to main memory and run forward and backward as separate kernels. Slang lets you fuse them. The SIGGRAPH neural shading courses ship mlp-training samples built on it, and the neural appearance paper this system follows was implemented in it.

WGSL has no autodiff. Neither does TSL. Neither does anything on the web platform.

So the derivatives in this codebase are written out by hand. Not because hand-writing them is better. Slang exists because it is worse. On the web there is no alternative. Which makes this a good place to see what autodiff does, since here it isn't hiding anything.

In-browser training is not exotic. webgpu-torch implements a PyTorch-shaped autograd library on WebGPU, tinygrad has a WebGPU backend, TensorFlow.js has had one for years. Any of those could train a network in a browser tab. What none of them will do is fuse forward, loss, and backward into a single dispatch that scatters gradients into a bilinearly-sampled texture, which is what this needs.

The loss is a tone curve#

Start with what we are minimising. Graphics people get this faster than ML people do.

Reflectance is high dynamic range. A specular peak can be hundreds of times brighter than the diffuse response beside it. Take a plain L1 or L2 loss on linear values and the peak dominates every gradient: the network spends its capacity on highlights and renders everything else as approximate mud.

Compare in a compressed space:

φ_p(x) = p · (x^(1/p) − 1),  with p = 3
loss   = |φ₃(ŷ) − φ₃(y)|

That is a tone curve. You already reach for one whenever you need to look at HDR data with human eyes; here we reach for one so that gradient descent can look at it with equally proportioned attention. Errors are measured perceptually rather than linearly, so a 5% error on a dim diffuse region matters as much as a 5% error on a bright highlight.

The cube-root specifically, rather than a log, because it stays finite and differentiable at zero. The derivative x^(−2/3) is large near zero but does not blow up the way 1/x does. In the code:

const predLog = pow( predClamped, float( 1.0 / 3.0 ) ).sub( 1.0 ).mul( 3.0 );
const refLog  = pow( refClamped,  float( 1.0 / 3.0 ) ).sub( 1.0 ).mul( 3.0 );
const diff    = predLog.sub( refLog );

Two losses#

The opacity head does not use this. Coverage is a probability in [0,1], not a radiance, so it gets a sigmoid output and binary cross-entropy:

const opPred = float( 1.0 ).div( float( 1.0 ).add( exp( val.negate() ) ) );
const bce = opTarget.mul( log( max( opPred, 1e-7 ) ) )
    .add( float( 1.0 ).sub( opTarget ).mul( log( max( float( 1.0 ).sub( opPred ), 1e-7 ) ) ) )
    .negate();

And its gradient is the reason to bother: for a sigmoid with BCE, ∂L/∂z collapses to pred − target. One subtraction. The exponentials cancel analytically. Pairing sigmoid with cross-entropy rather than squared error is worth doing for that, and the code shows it:

const delta_op = opPred.sub( opTarget ).mul( sampleWeight ).mul( invBatchUniform );

Emission uses the cube-root loss, like the BRDF. Three heads, two loss functions, all accumulating into one scalar.

The clamp that leaks#

Reflectance cannot be negative, so the output is clamped: ŷ = max(z, 0).

Earlier, in the MLP crash course, we noted that a clamp's derivative is a mask, and that a unit clamped to zero receives no gradient and can get stuck. For an output unit this is fatal. If the network ever pushes a colour channel negative early in training, that channel dies and the material loses a primary.

The mitigation is one constant:

const OUTPUT_CLAMP_GRADIENT_LEAK = 0.01;
const outputClampGradient = select( z3_c.greaterThan( 0.0 ), float( 1.0 ),
                                    float( OUTPUT_CLAMP_GRADIENT_LEAK ) );

One percent of the gradient gets through even when clamped. Enough for a dead output to climb back out; small enough not to distort normal training. Same idea as leaky ReLU, applied where it matters most.

The derivatives, written out#

Three MLP layers backward is mechanical, the pattern from the crash course, repeated. Receive delta from above, accumulate delta × activation into the weight gradient, push deltaᵀ W down, apply the ReLU mask:

const delta2_i = select( z2_i.greaterThan( 0.0 ), gradInput_i, float( 0.0 ) );

The frame construction is where it gets interesting, because normalize and cross are not operations you have ever differentiated by hand.

The Jacobian of normalize. If n = raw / |raw|, then:

∂L/∂raw = (g − n(n·g)) / |raw|

which is backwardNormalizeTSL in the source. Read it geometrically: the incoming gradient g is projected onto the plane perpendicular to n, and the component along n is discarded. That is correct because moving raw along its own direction does not change the normalised result at all. It only changes the length, which normalisation throws away. The 1/|raw| scaling says that a long vector is harder to rotate, so the same gradient produces a smaller angular change.

The backward of a cross product. b = n × t gives:

const gradNormN = gradN.add( cross( t, gradRawB ) );
const gradNormT = gradT.add( cross( gradRawB, n ) );

Each input's gradient is a cross product of the other input with the output gradient, with a sign convention that falls out of the antisymmetry. The frames are built normalize → cross → normalize, so the backward pass runs normalize → cross → normalize in reverse, composing all three.

There is no framework here. Ten lines of vector calculus, sitting in a compute shader, doing what a @differentiable annotation would have done for you.

Gradients that scatter into a texture#

The forward pass gathers a latent code by bilinear interpolation, four texels, weights w0 through w3:

const z_c = latents[off0 + c] * w0 + latents[off1 + c] * w1
          + latents[off2 + c] * w2 + latents[off3 + c] * w3;

The backward pass is the transpose of that gather, which is a scatter with the same weights:

atomicAdd( gradLatentsAtomic.element( off0.add( c ) ), int( gradZ_c.mul( w0 ) . . . ) );
atomicAdd( gradLatentsAtomic.element( off1.add( c ) ), int( gradZ_c.mul( w1 ) . . . ) );
atomicAdd( gradLatentsAtomic.element( off2.add( c ) ), int( gradZ_c.mul( w2 ) . . . ) );
atomicAdd( gradLatentsAtomic.element( off3.add( c ) ), int( gradZ_c.mul( w3 ) . . . ) );

A texel that contributed 70% of the sampled value receives 70% of the error. If a sample lands on a texel centre, that texel gets everything and its neighbours get nothing.

This is the moment the latent-texture claim becomes concrete. The texture is a parameter tensor, and this is the code that trains it: gradient flowing backwards through bilinear interpolation into individual texels of a mip level. Every training sample updates four texels of one level, weighted by how much each one mattered.

These are atomic adds. Thousands of samples are in flight, many landing on the same texel, and there is no ordering between them.

Fixed point, because WebGPU has no float atomics#

WGSL supports atomicAdd on integers. Not on floats. That is a hard platform limitation with no workaround.

So gradients are scaled and accumulated as integers:

atomicAdd( gradWeightsAtomic.element( idx ), int( gradW.mul( float( FIXED_POINT_SCALE ) ) ) );

and divided back down when read:

const rawGrad = float( atomicLoad( gradWeightsAtomic.element( idx ) ) ).div( float( FIXED_POINT_SCALE ) );

Fixed point rather than floating point, with FIXED_POINT_SCALE setting where the binary point sits. The trade is the usual one: absolute precision instead of relative, so very small gradients quantise to zero and very large sums can overflow. The scale has to be chosen so that a full batch's accumulated gradient fits in a 32-bit integer while individual contributions still register.

There is a second constant, GRADIENT_NORM_SCALE, used for the squared-norm accumulator. Squared gradients occupy a different magnitude range from gradients, so they need their own scale factor. Two accumulators, two fixed-point formats.

Integer atomic addition is associative, so training is bit-for-bit reproducible regardless of the order in which samples land. Float atomics, had they existed, would not have given you that.

Recomputing instead of remembering#

The backward pass through the frames needs rawN and rawT, the pre-normalisation vectors. The forward pass computed them.

It did not keep them. The backward pass computes them again, running the same eight-multiply-accumulate loop a second time.

This is activation checkpointing, and it is a deliberate trade. Storing them would cost 6 floats × batch size × mips of extra storage buffer, and every write and read is bandwidth. Recomputing costs 48 multiply-adds per frame per sample, which on a GPU running thousands of invocations concurrently is close to free. The arithmetic units are idle waiting on memory anyway.

On a GPU, ALU is cheap and bandwidth is expensive, and the ratio has been getting worse for twenty years. Large-scale training frameworks make this exact trade at a much larger granularity, recomputing whole transformer blocks rather than storing their activations. Same reasoning, six orders of magnitude apart.

Five dispatches per iteration#

The training loop, from NeuralAppearanceTrainer.train():

renderer.compute( trainBatchNode );
renderer.compute( resetGradientNormNode );
renderer.compute( accumulateGradientNormNode );
renderer.compute( adamWeightsNode );
renderer.compute( adamLatentsNode );

1. trainBatch: one invocation per sample. Fetch latents, build frames, forward through the MLP, compute loss, backward, atomically accumulate every gradient. Everything fused into one kernel, which is the thing frameworks are bad at.
2. resetGradientNorm: clear a single scalar. One invocation.
3. accumulateGradientNorm: one invocation per parameter, summing g² across both weights and latents into that scalar.
4. adamWeights: clip by the global norm, then step. One invocation per weight.
5. adamLatents: same, for the latent grid.

The norm needs its own pass because gradient clipping scales every gradient by min(1, maxNorm / ‖g‖), and you cannot know ‖g‖ until every sample has contributed. Within a single dispatch there is no barrier that spans workgroups. Invocations in different workgroups have no defined ordering and no way to wait for each other. The only global barrier available is the end of a dispatch. So: accumulate, end, reduce, end, apply.

Adam itself is unremarkable once you are here, momentum, second moment, bias correction, step, and it runs identically over both parameter buffers. That symmetry is the point. The latent texture is not special. It is a tensor of parameters, and it gets the same optimiser as everything else.

One last detail: the learning rate follows a cosine schedule down to cosineAnnealingScale (1%) of its initial value, so training starts by moving aggressively and finishes by polishing.

Press train#

The demo writes itself: a material, a loss curve, and a button.

That is the thing none of the reference implementations can offer. Slang's samples need Vulkan or Metal, a build toolchain, and on some paths a specific vendor's hardware. This runs in a tab. You can watch a neural material converge on your phone.

The runtime#

Training is over. What ships is a JSON file, two half-float textures with their mip chains, a hundred and eight frame weights, and about eighteen hundred MLP parameters, plus a NodeMaterial that knows how to evaluate them.

The material slots into the standard forward pass like any other. You can use a trained material without touching any of the training machinery above.

Picking a mip level by hand#

Ordinary textures get their mip level from the hardware. Here the latents are fetched, so the LOD calculation is explicit too:

footprint = max(‖∂UV/∂x · size‖, ‖∂UV/∂y · size‖)
lod       = clamp(log₂(footprint), 0, mips − 1)

dFdx and dFdy on the UVs, scaled by texture size, giving the number of texels this pixel covers. Familiar arithmetic. It is what the texture unit does internally, arriving somewhere unfamiliar, because the thing being selected is a learned encoding rather than a filtered colour.

The material supports a lodMode of trilinear, which does what you would hope:

const continuousLod = computeContinuousLOD( material, uvNode );
const baseMip = TSL.floor( continuousLod );
const fracMip = TSL.fract( continuousLod );
const nextMip = TSL.min( baseMip.add( 1 ), data.mipLevels - 1 );
// ... evaluate at both, then:
return TSL.mix( rgb0, rgb1, fracMip );

Note where the mix happens: after both levels have been decoded, not before. You cannot interpolate the latents between mip levels and decode once. Latent space is not linear, and a code halfway between two valid codes is not the material halfway between them. You decode twice and blend the results, which are radiances, and radiances do interpolate.

It costs a second full decode on the transition band. That is the price of a representation whose intermediate values have no meaning.

Fetching#

Two RGBA16F textures, bilinearly sampled at the selected level, giving eight channels. Half-float is sufficient because the latents were quantised to half during training. The network was fit in the presence of that quantisation rather than having it imposed afterwards, so the error is absorbed rather than accumulated.

Sixteen bytes per texel at base resolution, with a full mip chain, and that is the entire material.

Rebuilding the frames#

Same construction as before, W_n z + [0,0,1], W_t z + [1,0,0], normalize, cross, now in a fragment shader, from a 108-element uniform.

Getting there requires care about the incoming frame. transformToCanonicalFrame takes the TBN, re-orthogonalises the tangent against the normal by Gram-Schmidt, falls back to a synthesised tangent when the projection degenerates, and restores handedness by checking against the original bitangent:

const projectedTangent = frame[ 0 ].sub( normal.mul( frame[ 0 ].dot( normal ) ) );
// ...
const handedness = unhandedBitangent.dot( frame[ 1 ] ).lessThan( 0 ).select( - 1, 1 );

The comment in the source explains why: the derivative-based TBN fallback uses a shared scale for both axes, so a material trained in a canonical orthonormal frame would see different directions at runtime than it saw during training. A frame mismatch of a few degrees shows up as highlights in the wrong place, and it is the kind of bug that looks like a training failure when it is a coordinate convention failure.

The runtime frame must match the training frame. The network learned a function of projected directions. Project them differently and you are querying it off-distribution.

The cost#

The decoder evaluates as packed dot products against vec4 uniforms. At the default hidden size of 32:

Head	Cost	Rate
BRDF (20→32→32→3)	~1,800 MACs	per light, per fragment
Emission (8→3 linear)	24 MACs	per fragment
Opacity (8→1 linear)	8 MACs	per fragment

Fixed. It does not matter how complicated the source MaterialX graph was.

Two notes on that table. The BRDF is per light because it depends on wi. The aux heads are not, because emission and coverage depend only on the latent code, so they are evaluated once per fragment regardless of how many lights are in the scene. And the whole thing is a ceiling, not an average: a fixed cost you can put in a budget, which a node graph never was.

Slotting into the lighting model#

return applyOutputActivation( decoded, brdf.outputActivation ).mul( wi.z.max( 0 ) );

That is the end of evaluateNeuralBRDF. Decode, then multiply by the cosine, which training divided out earlier, closing the loop. The caller multiplies by light colour and intensity.

This lives in NeuralAppearanceLightingModel.direct(), which is called once per punctual or directional light in the standard forward pass. The consequences are all of the "nothing happens" variety, and that is the good news:

• Shadow maps work. Depth prepass works. Tone mapping works.
• Emission goes to emissiveNode, opacity to opacityNode and alphaTestNode, so cutout silhouettes integrate with depth and shadow passes for free.
• It is a NodeMaterial. Assign it to a mesh. There is no separate pass, no custom renderer, no scene graph requirement.

The NVIDIA version of this technique needs eval(), sample() and pdf(), because a path tracer must generate scattered rays and know their probability density. Rasterising against punctual lights, wi is given to you by the scene. Only eval() is needed, which is why two thirds of the reference architecture is absent here.

Back to where we started#

Load the opening comparison again. Same asset, same lighting, camera walking back.

The neural material holds. The highlights stay highlights, the fine structure prefilters into something that still looks like the material, and there is no shimmer band in the mid-distance.

Three mechanisms are responsible:

• Mip levels trained rather than filtered, so no level contains an average of parameters that could not be averaged.
• Learned frames absorbing mesostructure, so normal-map detail is part of a representation that was optimised to prefilter.
• Footprint-filtered training targets, so every level was fit against a correct integral over the area it covers.

The correct average of two materials is the average of their appearance. If you want a representation that survives filtering, you have to train it against the thing you want.

What it can't do, and what's next#

Limits of the current implementation. Most of them have an obvious shape: you can see what the fix looks like. It is not built.

Direct lighting only#

The runtime evaluates the neural BRDF against punctual and directional lights. There is no image-based lighting and no spherical harmonic probe support.

A punctual light hands you wi; you call eval() and you are done. An environment map hands you a whole hemisphere of incoming radiance, and getting a reflectance out of that requires an integral. In a conventional pipeline you dodge this with split-sum prefiltering: precompute the environment convolved against a family of GGX lobes, index by roughness. But there is no roughness parameter here to index by. There are eight latent channels whose relationship to lobe width is whatever training decided.

The query parameters have to become network outputs. A small head predicting a dominant direction, a lobe width, and a scale/bias pair would let the standard split-sum machinery apply, and would absorb the DFG lookup table into the network, since that table is itself a tabulated function of BRDF statistics.

Two routes from there:

Moment supervision, environment-independent. Integrate the teacher over wi to get directional albedo and the first moment of the lobe, and supervise the head against those. Portable, no environments in the training loop.

End-to-end against an environment bank, which lets the network absorb split-sum error rather than inherit it, at the cost of prefiltering inside the training loop.

There is a useful energy identity either way. Under a uniform white environment every prefiltered sample equals 1 and irradiance equals π, so the head's outputs must satisfy diffuseAlbedo · π + specScale + specBias == ρ, where ρ is the measured directional albedo. A white-furnace test, cheap to compute, and a loss term worth having.

This is the largest gap.

Sharp spatial discontinuities blur#

Train a material with a hard black-and-white checkerboard and the neural version has a soft ramp where the source has an edge. This is the most visible quality limitation in the system. At least three different things are going on, and they have different fixes.

The sample budget is thinner than it looks. generateTrainingSamples places UVs on a stratified jittered uniform grid, and each sample is assigned one exponentially-drawn mip level. Work through the arithmetic at the defaults, batchSize: 1024, 512² latents, 2,000 iterations, and mip 0 receives on the order of a dozen samples per texel over the entire training run, each at a different random (wi, wo) pair.

That is a 4D directional function per texel, fit from a dozen observations. Everything else comes from smoothness and from sharing structure across neighbouring texels. Under-determined fits produce smooth answers. That is what least-error estimation does when the evidence runs out. The blur is the correct response to insufficient data.

The sampler has no spatial importance sampling. Earlier we made a careful argument for importance-sampling directions: uniform hemisphere sampling starves a specular peak, so the mixture concentrates queries where the function has structure.

None of that reasoning has been applied to the spatial domain. UVs are drawn area-uniformly, so an edge receives samples in proportion to its area, which is nearly zero. The exact failure the direction sampler was built to avoid is still present one axis over.

The fix is the same shape: precompute the teacher's spatial gradient magnitude, build a CDF over it, and draw UVs from a mixture of uniform and gradient-weighted distributions with MIS weights to stay unbiased. Edges get one to two orders of magnitude more samples at no extra per-iteration cost.

Interpolation smooths in both directions. Two mechanisms push toward smoothness independent of sample count.

Forward, you interpolate then decode. Bilinear happens in latent space, and a code halfway between two texels is not the code for the appearance halfway between them. Latent space is not linear. The runtime avoided this for mip transitions by decoding both levels and blending the results, but that fix is unavailable here, because the four bilinear taps are the sampling mechanism.

Backward, the gradient scatter diffuses. Every sample deposits gradient into all four taps weighted by the same bilinear weights. A sample near an edge pushes its side's gradient into texels on the other side. Repeated every iteration, that is a low-pass filter applied to the latent grid, and nothing counteracts it except sample density.

Things worth trying, in order of leverage per unit effort:

1. Edge-aware UV importance sampling, as above. The structural fix.
2. More samples where the detail is. Raise iterations and batchSize, and tune the mip distribution to concentrate on level 0. If a tenfold increase sharpens edges, the problem is convergence rather than capacity, which is good news.
3. A gradient-domain loss term. Query the teacher at uv ± ε, and match the student's finite-difference spatial derivative to the teacher's. The current loss is point-wise and therefore indifferent between a sharp edge and a smooth ramp with the same mean error. This penalises blur directly.
4. Learnable interpolation sharpness. One trained parameter k, sharpening the fractional coordinates before the bilinear weights are computed: t' = clamp((t − 0.5)·k + 0.5, 0, 1). At k = 1 it is current behaviour; larger values compress the transition into a narrower band. Trivial backward pass, and it lets each material choose its own sharpness.
5. Higher latent resolution. Works, and is the obvious move, but be clear about why: it does not make the transition sharper in texel space. That stays about one texel wide. It makes texels smaller. Cost is quadratic in memory, tripled again by Adam's moment buffers.

A useful diagnostic before any of that: check whether cutout alpha stays sharp on the same asset. The opacity head ends in a sigmoid, and a saturating output manufactures sharpness that a linear output cannot. If alpha is crisp while colour is soft, the latent grid is fine and the problem lives in the BRDF output path. If both blur, it is the grid, and it is a sampling or resolution problem.

Rule out the honest cases first. At any distance, a checkerboard near the grid's texel frequency should decode to grey. That is prefiltering working, not a defect. If the checkerboard's period is not an integer multiple of texel spacing, no amount of training will produce a sharp edge, because the grid cannot represent the edge position. That is the Nyquist limit, the same limit an ordinary texture has, and it is not neural-specific.

[TODO: this section states a diagnosis that has not yet been measured. Run the sample-budget experiment and the alpha-sharpness check before publishing, and replace the hypothesis with whatever the numbers actually say.]

Opaque and cutout only#

Transmission, glass, water, anything with a BTDF, is outside the representation, as is subsurface scattering and blended fractional alpha.

Cutout works because coverage is a scalar per texel that the opacity head can learn. Transmission does not, because it doubles the direction domain: the teacher would have to sample outgoing directions below the surface, the decoder would need to represent a discontinuity at the boundary, and the whole thing interacts with sorting and depth in ways cutout does not.

The 20-input decoder layout has no room reserved for a transmission flag either, so this is a format change, not only a training change.

No analytical parameter encoder#

The reference implementation pre-trains an encoder mapping analytical material parameters, albedo, roughness, metallic, to latent codes, then fine-tunes. This implementation optimises random latents from scratch, which needs more iterations to resolve spatial patterns that an encoder would have produced immediately.

For a single material this is a training-time cost and nothing more. For an asset library it is the difference between two architectures.

Recall the parameter accounting from earlier: ~1,950 network parameters against millions of latent values. The MLP is a shared codec. If it were shared in fact, one decoder trained across a corpus, per-material latent grids only, you would ship about two kilobytes of weights once for an entire catalogue, and each material would cost only its texture.

The format already separates these. Nothing in the runtime assumes the decoder is unique to the asset. What is missing is the training procedure: a corpus, a schedule that trains the shared decoder across materials while latents stay per-material, and a way to fit new latents against a frozen decoder. That last one is the interesting piece, because it is gradient descent on the latent grid with the weights held fixed. The code path is nearly there already.

This is the most valuable unbuilt thing in the system.

Training is still bounded#

Training runs on the GPU now. The CPU path has been removed, and NeuralAppearanceTrainer throws if handed a non-WebGPU renderer. But high-resolution latent grids are still constrained by what the browser will let you allocate.

Adam is the reason the number is worse than you expect. Each parameter needs its value plus two optimiser moments, so a 2,048² latent grid costs three times its storage size during training. estimateTrainingMemory() will tell you the number for a given resolution; check it before starting a long run rather than after.

What would help most#

If you want to work on this, in rough order of value:

1. Edge-aware UV importance sampling. Smallest change with the most visible quality return, and it fixes a gap the direction sampler already showed us how to think about.
2. Shared decoder + latent-only fitting. The library architecture described above. Highest architectural leverage.
3. IBL, by either route. The largest capability gap.
4. A parameter encoder, which reduces training time for everyone and is a prerequisite for doing (2) well.
5. Better validation metrics. NeuralAppearanceValidator checks angular bins, reciprocity and smoothness, but there is no perceptual metric, no edge-sharpness metric, no white-furnace energy check, and no automated regression suite across a material corpus.

[TODO: this section needs measurements, not estimates. Required before publishing:

• Training time at 256², 512², 1024², on at least two GPU classes
• Asset size vs. the source MaterialX texture set, same materials
• Frame cost vs. MeshPhysicalMaterial with the equivalent graph, same scene
• Samples-per-texel at mip 0 for the default settings, and an edge-sharpness comparison across sample budgets
• A gallery of failure cases: the checkerboard above, and whatever else falls over]

Publish the cases where it loses. Readers will look for them regardless, and finding them yourself is what makes the wins credible.

Where the code is#

• examples/jsm/neural/: the implementation
• examples/jsm/loaders/NeuralAppearanceLoader.js: asset loading and validation
• examples/webgpu_materials_neural_appearance.html: the viewer and training UI
• utils/neural-appearance/convert_checkpoint.py: bridge for official NVIDIA checkpoints

The format is versioned and documented in NeuralAppearanceFormat.js. Issues and pull requests welcome, particularly on the five items above.