Gaussian Descent

The Spectrum of Descent

The representations used in 3D reconstruction form a spectrum, ordered by how much of the base object $X$ they actually materialize:

The mesh is $X$ given a CW-complex structure — combinatorial, rigid, with explicit topology. The point cloud discards the combinatorics but keeps the discrete samples. The 3D Gaussian replaces discrete points with continuous distributions. And the NeRF abandons the base entirely, storing only the functor. Each step trades geometric commitment for representational flexibility.

3D Gaussians as Measures on $X$

Each 3D Gaussian is defined by five attributes: a position $\boldsymbol{\mu} \in \mathbb{R}^3$, a covariance $\boldsymbol{\Sigma} \in \text{Sym}^+_3(\mathbb{R})$ (encoded as a rotation quaternion $\mathbf{q}$ and scale vector $\mathbf{s}$), an opacity $\alpha \in [0,1]$, and color coefficients in spherical harmonics $\{c_\ell^m\}$.

The full scene is a mixture of measures: $\mu_{\text{scene}} = \sum_{k=1}^{N} \alpha_k \, \mathcal{N}(\boldsymbol{\mu}_k, \boldsymbol{\Sigma}_k)$. This object lives on $X$ — it has descended from the views into 3D space. But it is not a submanifold. It is a distribution in the measure-theoretic sense: a weighted sum of Gaussians that approximates the density of the scene.

Notice how the Gaussians cluster on surfaces, edges, and detail regions. They are not uniformly distributed — the optimization process (which we’ll see shortly) adaptively densifies them where the scene has high-frequency detail and prunes them where the scene is smooth. This is the measure adapting its support to the geometry of $X$.

Splatting as Pushforward $\pi_{i*}$

To render a view from the Gaussians, you project each 3D Gaussian through the camera. A 3D Gaussian $\mathcal{N}(\boldsymbol{\mu}, \boldsymbol{\Sigma})$ under an affine projection $\pi_i$ becomes a 2D Gaussian $\mathcal{N}(\pi_i(\boldsymbol{\mu}), J \boldsymbol{\Sigma} J^\top)$ where $J$ is the Jacobian of the projection.

This is the pushforward of measures: $\pi_{i*}\mu = \mu \circ \pi_i^{-1}$. In Part I we saw the pullback $\pi_i^*$ — the camera lifting structure from the base to a flat image. Now we go the other direction. The pushforward takes the 3D measure and splats it onto the image plane.

where $W$ is the world-to-camera transform and $J$ is the projection Jacobian. The 3D ellipsoid becomes a 2D ellipse on screen — the "splat."

The Differentiable Loop: Gradient Flow on the Descended Object

Here is where 3D Gaussian Splatting diverges from classical photogrammetry. In the traditional pipeline, you reconstruct once: establish correspondences, triangulate, adjust, done. In 3DGS, reconstruction is a continuous optimization — a gradient flow on the space of measures.

The loss function is the photometric error between the rendered image and the ground-truth photograph:

Because the entire rendering pipeline — projection, splatting, alpha compositing — is differentiable, gradients flow back through the image into every Gaussian parameter: $\nabla_{\boldsymbol{\mu}} \mathcal{L}$, $\nabla_{\boldsymbol{\Sigma}} \mathcal{L}$, $\nabla_\alpha \mathcal{L}$, $\nabla_{\text{SH}} \mathcal{L}$.

The adaptive density control is the crucial mechanism. Gaussians in regions of high positional gradient — where the current rendering poorly matches the ground truth — get split (large Gaussians) or cloned (small ones) to increase detail. Gaussians with near-zero opacity get pruned. The measure adapts its own support during optimization.

Alpha Compositing: The Volume Rendering Integral, Discretized

Once the 2D splats are computed, they must be composited into a final pixel color. This is done by sorting the Gaussians by depth and blending front-to-back:

This is the discrete analogue of the volume rendering integral from NeRF:

The NeRF evaluates this integral by marching along each ray and sampling the density field at discrete points. The Gaussian splatter achieves the same result by projecting each primitive to 2D and blending analytically. Same integral, different computational strategy — and the Gaussian version is orders of magnitude faster.

Real-Time Rendering: The Functor Evaluated at 60fps

The practical miracle of 3D Gaussian Splatting is that the entire pipeline — sort by depth, project to 2D ellipses, alpha-composite front-to-back — runs in real time. Not "research demo" real time. Not "if you have an A100" real time. Consumer GPU, 60+ frames per second, 1080p, real time.

What this means, in the descent vocabulary: you can evaluate the functor of views at interactive rates. Pick any probe $T$ — any camera position and orientation — and the pushforward $\pi_{T*}\mu$ is computed and displayed before the next frame. The sheaf answers queries in real time.

Each frame you see is a new element of $h_X(T) = \text{Hom}(T, X)$. The 3D Gaussians are the descended object $\mu$ on $X$; the rendering pipeline computes $\pi_{T*}\mu$ for your current camera $T$; and the pixels on screen are the answer. You are querying the Yoneda functor at 60fps.

The Complete Hierarchy

We can now place every representation in the taxonomy of descent. The question is always the same: how much of $X$ do you materialize?

The spherical harmonics deserve a footnote. The color of each Gaussian is not a fixed RGB value — it is a function on the sphere $S^2$ of viewing directions, encoded as spherical harmonic coefficients up to degree $\ell$. This means the color is itself a small sheaf: $\mathbf{d} \mapsto c(\mathbf{d})$. The descended object carries, at each point, a local version of the NeRF functor. The measure descends. The color stays a sheaf.