What’s inside a NeRF?

(or actually: what’s it inside of?)

Simply put, a “Radiance Field” takes 2D images of a place and turns them into a 3D model of that place. Typically, it is created by feeding 2D footage of a physical environment into a feature detection and triangulation algorithm that converts the frames into a 3D representation of the light in the original scene as captured by the input video.

Its ability to easily encapsulate the appearance of a place and to efficiently interpolate new realistic views of it serves the needs of motion picture VFX beautifully but, as is often the case in many other domains, it is in the transition areas that we end up finding the most intriguing artifacts. Renders of radiance fields have a tendency to break beautifully in the areas where the algorithm doesn’t have enough input data to infer realistic views of the scene. Recognizable features begin to tear into sharp colorful patterns, dissolve into soft wispy shapes, and eventually fade into the background.

What we end up with is an abstract structure with a precise and tangible center cocooned inside a shell of decreasing density that eventually fades into nothingness. An intense memory from a dream: there is a vivid core, but the visceral feeling it holds never manages to become anchored. The place where it manifests is murky and elusive. It’s absolutely real and tangible but beyond language. A flood of sensory input has encoded it into a place that holds the potential to recreate the scene. A memory is a dimensional network of neurons connecting through specific sequences of electrical impulses that we experience as a marker of meaning, with fraying edges that decompose into a seemingly infinite mystery.