Adding View-Dependent Gaussian Splats to ThreeJS

When I added native Gaussian Splatting support to Three.js, the first renderer handled position, covariance, color, opacity, sorting, and alpha blending. It rendered many captures well, but it treated color as fixed from every camera angle.

I have now added view-dependent color in Three.js PR #34215. The PR adds spherical harmonics (SH) loading and rendering for Gaussian splats, supporting SH0, SH1, SH2, and SH3.

The Lion splat shows the change. With flat color, the back side of the model stays dull as the camera moves. With SH3 enabled, a white rim and sheen-like fringe appears around the surface when you orbit into the right angle.

Why View Dependence Matters#

In the original 3DGS training pipeline, each Gaussian can store color that changes with view direction. A single RGB value loses that information.

You see view dependence around object silhouettes as Fresnel-like brightening or sheen. Specular highlights slide across glossy surfaces. Gemstone captures change as facets pick up different reflections. Even diffuse-looking captures often contain directional color shifts from the source photos.

Flat color, equivalent to SH0, bakes all of that into one RGB value per splat. It can look right from one angle and wrong from another. Spherical harmonics give each splat a compact angular color function.

The renderer still uses captured appearance, not a physical reflection model. It takes the view direction from the camera to the splat, evaluates a small SH basis, multiplies those basis values by the captured RGB coefficients, and adds that contribution to the base color.

Three.js evaluates the SH color in the vertex node, using the view direction from the camera to the splat center. The four vertices of a splat quad compute the same directional color, so this is effectively per splat rather than per fragment. (This is the same approach taking the Spark library.)

Spherical Harmonics Packing#

Spherical harmonics are basis functions over directions on a sphere. For Gaussian splats, they store how a splat's color changes as the camera moves around it.

The order controls how much directional detail the splat can store:

Order	Coefficients in order	Total coefficients
SH0	1	1
SH1	3	4
SH2	5	9
SH3	7	16

Each coefficient is an RGB vector. Three.js stores SH0 in the regular color attribute, then stores the higher-order SH1-SH3 coefficients in sphericalHarmonics1, sphericalHarmonics2, and sphericalHarmonics3.

This can add up to a lot of data. I chose a byte-aligned representation. Each higher-order scalar coefficient gets clamped into a byte, with decoding in the renderer using roughly:

coefficient = (byte - 128) / 128;

Four of those bytes get packed into one Uint32 word for upload to the GPU. The loaders write bytes into a Uint8ClampedArray view over the same buffer, then hand the Uint32Array to the geometry. The renderer unpacks each byte with shifts and masks.

Libraries such as the Spark Gaussian Splatting library use denser bit packing. Spark stores SH1 in 7-bit fields, SH2 in 8-bit fields, and SH3 in 6-bit fields. That saves memory, but the loader and shader both have to deal with narrower fields and values that can cross word boundaries.

The Three.js version spends a little more memory to keep loading and rendering dead simple:

SH order	Spark total	Three.js total	Three.js vs Spark
0	64	64	+0%
1	72	76	+5.6%
2	88	92	+4.5%
3	104	116	+11.5%

Code and Bundle Impact#

Measured against the PR merge base, the runtime addon changes add about 17.6 KB of raw source. For a typical bundle that exports GaussianSplatMesh, SPZLoader, KSPLATLoader, and GLTFGaussianSplatLoaderExtension, the additional cost for view-dependent Gaussian Splat loading + rendering is 2KB of compressed minified code - not bad in my opinion:

Target	Size
Raw source	+17,553 B
Minified	+6,026 B
gzip	+1,812 B
Brotli	+1,606 B

I measured that with Rollup and Terser, treating three, three/webgpu, and three/tsl as externals.