{"id":1299,"date":"2025-05-26T15:10:54","date_gmt":"2025-05-26T13:10:54","guid":{"rendered":"https:\/\/cammonte.com\/?page_id=1299"},"modified":"2025-12-08T14:53:21","modified_gmt":"2025-12-08T13:53:21","slug":"original-nerf-paper-breakdown","status":"publish","type":"page","link":"https:\/\/cammonte.com\/index.php\/original-nerf-paper-breakdown\/","title":{"rendered":"NeRF Basics"},"content":{"rendered":"\n

TL;DR<\/h1>\n\n\n\n

NeRFs represent a 3D scene as a fully-connected deep network<\/mark><\/strong><\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

Map any 3D location [math](x, y, z)[\/math] and viewing direction [math](\\theta, \\phi)[\/math] to a volume density value [math]\\sigma[\/math] and a colour (emitted radiance) [math]c=(r, g, b)[\/math] at that location that can then be used to render a novel view with classical volume rendering techniques<\/strong><\/p>\n\n\n\n

How it works<\/strong><\/h2>\n\n\n\n

Sample<\/h3>\n\n\n\n
\n
\n

Trace a camera ray [math]r(t)[\/math] through the currently shaded pixel into the scene and sample [math]s_0, s_1, …, s_N[\/math] along it, [math]s_i=(x_i, y_i, z_i)[\/math]<\/p>\n<\/div>\n\n\n\n

\n
\"\"<\/figure>\n<\/div>\n<\/div>\n\n\n\n

Infer<\/h3>\n\n\n\n
\n
\n

For each sample, have the trained network infer its corresponding density [math]\\sigma_i[\/math] and colour [math]c_i[\/math]<\/p>\n<\/div>\n\n\n\n

\n
\"\"<\/figure>\n<\/div>\n<\/div>\n\n\n\n

Accumulate<\/h3>\n\n\n\n
\n
\n

Use classical volume rendering to accumulate colours and densities into the colour of the shaded pixel<\/p>\n<\/div>\n\n\n\n

\n

[math] C(r) = \\int_{t_n}^{t_f} [\/math] [math] T(t) [\/math]<\/mark> [math] \\sigma(r(t)) [\/math]<\/mark> [math] c(r(t), d) [\/math]<\/mark> [math] dt [\/math]<\/p>\n\n\n\n

\n
\n

Accumulated transmittance from [math]t_n[\/math] to [math]t[\/math]<\/mark><\/strong><\/p>\n<\/div>\n\n\n\n

\n

Volume density at [math]r(t)[\/math]<\/mark><\/strong><\/p>\n<\/div>\n\n\n\n

\n

Colour at [math]r(t)[\/math] when looking in the direction of [math]d[\/math]<\/mark><\/strong><\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n

Backpropagate<\/h3>\n\n\n\n

TODO<\/mark><\/strong><\/p>\n\n\n\n

Why it works<\/h2>\n\n\n\n
    \n
  • Volume rendering is naturally differentiable (it’s literally the result of an integral) <\/strong>\u2192 use gradient descent to train the model by minimising error between observed images of the scene and corresponding rendered views<\/li>\n\n\n\n
  • Positional encoding<\/strong>\n
      \n
    • Issue<\/strong>: basic implementation of the above does not capture high frequency details (lots of sharp changes over a small area of the image) because the input is only 5D<\/li>\n\n\n\n
    • Solution<\/strong>: project the 5D input into a higher dimensional space through sin and cos functions to represent both coarse and fine details<\/li>\n<\/ul>\n<\/li>\n\n\n\n
    • Hierarchical volume sampling<\/strong>\n
        \n
      • Issue<\/strong>: basic implementation of the above requires a lot of samples per camera rays to accurately capture a scene since they are sampled at random<\/li>\n\n\n\n
      • Solution<\/strong>: use two networks: a coarse one that takes as input samples taken coarsely at random along the ray, use the densities it outputs to drive a finer sampling that will serve as input to a fine network which will give the final output densities and colours used for rendering <\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n\n

        Why it’s cool<\/h2>\n\n\n\n
          \n
        • Can model complex geometry and non-Lambertian surfaces <\/strong>(colour of the surface changes depending on the viewing direction)<\/li>\n\n\n\n
        • Gives an extremely compact representation of a 3D scene<\/strong>: NeRF optimised weights need less memory than the input JPEG images it trained on<\/li>\n\n\n\n
        • Gives a continuous representation of a 3D scene<\/strong> (prior related work is discrete)<\/li>\n<\/ul>\n\n\n\n

          Related Work<\/h1>\n\n\n\n

          Implicit 3D shape representations<\/h2>\n\n\n\n

          Represent continuous 3D shapes implicitly through functions mapping any spatial points to some meaningful value<\/mark><\/strong><\/p>\n\n\n\n

          Using ground truth geometry<\/h3>\n\n\n\n

          How it works<\/h4>\n\n\n\n

          Optimise a network to map any spatial point [math]xyz[\/math] to:<\/strong><\/p>\n\n\n\n

            \n
          • Signed distance functions<\/strong>: how far that point is from the closest surface of the shape\n
              \n
            • Minimise MSE loss between predicted and ground truth values at sampled 3D coordinates (regression)<\/li>\n\n\n\n
            • SDF are continuous and differentiable so we can optimise on them<\/li>\n\n\n\n
            • Surface can be extracted via marching cubes<\/strong> (SDF = 0 for points on the surface)<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n\n
                \n
              • Occupancy fields<\/strong>: maps [math]xyz[\/math] to [math]\\sigma \\in [0, 1][\/math] indicating occupancy probability of the point (whether it’s inside the shape (1) or outside (0))\n
                  \n
                • Minimise a binary cross-entropy loss between predicted occupancy and ground truth label (binary classification, easier than regressing SDF: easier to define labels, more stable gradients, faster convergence)<\/li>\n\n\n\n
                • Surface can be extracted via marching cubes<\/strong> (occupancy probability is 0.5 for points on the surface)<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n
                  \n
                  \n\t\t\t
                  \n\t\t\t

                  Marching cubes algorithm<\/strong><\/p>\n\t\t\t

                  <\/span><\/div>\n\t\t<\/div>\n\t\t\t
                  \n\n

                  Convert an implicit surface into a polygonal mesh<\/strong><\/p>\n\n\n\n

                    \n
                  • Sample the 3D space at regular intervals<\/li>\n\n\n\n
                  • Evaluate the function representing the implicit surface (SDF, occupancy field, …) at each grid point<\/li>\n\n\n\n
                  • Identify grid cells where the function value crosses the threshold value (0 for SDF, 0.5 for occupancy field) across corners (means the surface passes through that cell)<\/li>\n\n\n\n
                  • Approximate the surface within these cells according to a precomputed lookup table\n
                      \n
                    • For each cube in the grid, you have 8^2=256 possible configurations (SDF\/occupancy for each of the 8 corners is over or below the threshold) and for each configuration you use a precomputed lookup table to know how to draw triangles within that cell to approximate the surface<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n<\/div>\n\t\t<\/div>\n<\/div>\n\n\n

                      Limitations<\/h4>\n\n\n\n
                        \n
                      • Requires ground truth geometry to optimise on<\/li>\n<\/ul>\n\n\n\n

                        Leveraging differentiable rendering functions<\/h3>\n\n\n\n

                        How it works<\/h4>\n\n\n\n

                        Formulate differentiable rendering functions to use the above methods with ground truth images rather than ground truth geometry<\/strong><\/p>\n\n\n

                        \n
                        \n\t\t\t
                        \n\t\t\t

                        Numerical method and implicit differentiation example<\/strong><\/p>\n\t\t\t

                        <\/span><\/div>\n\t\t<\/div>\n\t\t\t
                        \n\n
                          \n
                        • Cast a camera ray [math]r(t) = o + td[\/math] through the image plane into the 3D scene and numerically search for the point [math]t*[\/math] where the ray intersects the implicitly defined surface (ex: for application to the occupancy fields [math]f(r(t*))=0.5[\/math])<\/li>\n\n\n\n
                        • Once you find [math]t*[\/math], you ask the network to predict the colour at that point, compare to the ground truth colour of the image, and then you want to backpropagate through this loss [math]L[\/math] to update the neural network<\/li>\n\n\n\n
                        • The iterative numerical method used to find t* is not differentiable, so use implicit differentiation<\/strong> on the constraint [math]f(r(t*))=0.5[\/math] to compute [math]\\frac{dL}{d \\theta}[\/math]<\/li>\n<\/ul>\n\n<\/div>\n\t\t<\/div>\n<\/div>\n\n\n

                          Limitations<\/h4>\n\n\n\n
                            \n
                          • Limited to simple shapes with low geometric complexity \u2192 over smoothed renderings<\/li>\n<\/ul>\n\n\n\n

                            View synthesis and image-based rendering<\/h2>\n\n\n\n

                            Synthesise high-quality photorealistic novel views of a scene from a set of input RGB images of that scene<\/mark><\/strong><\/p>\n\n\n\n

                            Light field sample interpolation<\/h3>\n\n\n\n

                            How it works<\/h4>\n\n\n\n
                              \n
                            • The light field represents all the light rays in a scene: colour and intensity as a function of pixel position and viewing direction<\/li>\n<\/ul>\n\n\n\n
                              \n
                              \n

                              [math]L(u, v, s, t)[\/math]<\/p>\n<\/div>\n\n\n\n

                              \n
                                \n
                              • [math](u, v)[\/math]: image coordinates<\/li>\n\n\n\n
                              • [math](s, t)[\/math]: viewing directions<\/li>\n<\/ul>\n<\/div>\n<\/div>\n\n\n\n
                                  \n
                                • To generate light field values for unseen [math](s, t)[\/math] pairs, we interpolate colour and intensity values for the unseen pair from nearby sampled views to generate a novel view<\/li>\n<\/ul>\n\n\n\n

                                  Limitations<\/h4>\n\n\n\n
                                    \n
                                  • Requires dense and regularly spaced input views<\/li>\n\n\n\n
                                  • Assumes scene continuity: locally smooth, surfaces and colours change gradually across views \u2192 miss high frequency details and sharp edges<\/li>\n<\/ul>\n\n\n\n

                                    Scene Representation Networks<\/h3>\n\n\n\n

                                    How it works <\/strong><\/h4>\n\n\n\n

                                    Represent scenes as continuous functions that map world coordinates to a feature representation of local scene properties<\/strong><\/p>\n\n\n\n

                                    For each pixel in the novel view to be rendered:<\/p>\n\n\n\n

                                      \n
                                    • Trained depth MLP<\/strong> predicts depth at which the ray hits a surface: takes as input ray origin and direction, camera parameters and scene code<\/strong> (latent vector representing the current scene) and outputs a predicted t (ray paramter) where closest intersection occurs\n
                                        \n
                                      • Can train an input images to scene code encoder -> SRN is generalisable across different scenes<\/strong><\/li>\n<\/ul>\n<\/li>\n\n\n\n
                                      • Trained scene MLP<\/strong> predicts RGB colour (and additional properties like occupancy, visibility, …) from intersection point in space and scene code<\/li>\n<\/ul>\n\n\n\n

                                        Limitations<\/strong><\/h4>\n\n\n\n
                                          \n
                                        • Learned ray marching is less stable than volume rendering used by NeRF<\/li>\n\n\n\n
                                        • View independent: does not consider viewing direction when predicting colour -> can’t predict specularities or reflections<\/li>\n<\/ul>\n\n\n\n

                                          Sampled volume representations<\/h3>\n\n\n\n

                                          Learn volumetric scene parameters at sampled points in the scene<\/strong><\/p>\n\n\n\n

                                          Colouring voxel grids<\/h4>\n\n\n\n

                                          Use observed images to directly colour voxel grids<\/strong><\/p>\n\n\n\n

                                            \n
                                          1. Define a regular grid of 3D voxels inside a 3D bounding volume that encloses the scene<\/li>\n\n\n\n
                                          2. For each voxel, use the known camera intrinsics and extrinsics to project it into each image, define the colour of this voxel as the average of the colours of all corresponding pixels over the input images<\/li>\n<\/ol>\n\n\n\n

                                            Limitations<\/strong>: assumes colour does not depend on viewing direction, need camera parameters<\/p>\n\n\n\n

                                            Sampled volume representations<\/h4>\n\n\n\n

                                            Train scene-independent deep network that predicts a sampled representation from a set of input images and use alpha compositing or learned compositing to render novel views<\/strong><\/p>\n\n\n

                                            \n
                                            \n\t\t\t
                                            \n\t\t\t

                                            Alpha compositing (classic volume rendering)<\/strong><\/p>\n\t\t\t

                                            <\/span><\/div>\n\t\t<\/div>\n\t\t\t
                                            \n\n

                                            Render volumetric representations (we have colour and density at any given point in space)<\/strong><\/p>\n\n\n\n

                                            Combine predicted colour and density samples through a fixed equation<\/strong><\/p>\n\n\n\n

                                            To render an image using predicted colour [math]c_i[\/math] and density [math]\\sigma_i[\/math] values at a given point and viewing direction, we do the following for each camera ray:<\/p>\n\n\n\n

                                              \n
                                            • Sample points along the ray in the 3D space<\/li>\n\n\n\n
                                            • Predict colour and density at these points<\/li>\n\n\n\n
                                            • Blend them using the standard volume rendering equation:\n
                                                \n
                                              • [math]C = \\sum_{i}T_i alpha_i c_i[\/math]\n
                                                  \n
                                                • Opacity [math]\\alpha_i = 1 – \\text{exp}(-\\sigma_i \\delta_i)[\/math]<\/li>\n\n\n\n
                                                • Transmittance [math]\\T_i=\\text{exp}(- \\sum_{j=1}^{i-1} \\sigma_j \\delta_i)[\/math]<\/li>\n\n\n\n
                                                • Distance between adjacent samples [math]\\delta_i[\/math]<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n<\/div>\n\t\t<\/div>\n<\/div>\n\n
                                                  \n
                                                  \n\t\t\t
                                                  \n\t\t\t

                                                  Learned compositing (neural volume rendering)<\/strong><\/p>\n\t\t\t

                                                  <\/span><\/div>\n\t\t<\/div>\n\t\t\t
                                                  \n\n

                                                  Render volumetric or sampled representations (we have value at any given point in space and viewing direction)<\/strong><\/p>\n\n\n\n

                                                  Train a network to combine them<\/strong><\/p>\n\n\n\n

                                                    \n
                                                  • More expressive than alpha compositing:\n
                                                      \n
                                                    • Can model non-Lambertian effects (reflections, specular highlights)<\/li>\n\n\n\n
                                                    • Can learn to handle occlusions, depth ambiguity<\/li>\n<\/ul>\n<\/li>\n\n\n\n
                                                    • Look more into it<\/mark><\/strong><\/li>\n<\/ul>\n\n<\/div>\n\t\t<\/div>\n<\/div>\n\n\n

                                                      Relevant example is “Neural Volumes: Learning Dynamic Renderable Volumes from Images” (SIGGRAPH 2019)<\/mark><\/strong><\/p>\n\n\n\n

                                                      Limitations<\/h4>\n\n\n\n
                                                        \n
                                                      • Can’t scale to higher resolution imagery because of poor time and space complexity due to discrete sampling\n
                                                          \n
                                                        • NeRF encodes a continuous<\/em> volume -> requires way less storage than sampled volumetric representations<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n\n

                                                          Architecture<\/h1>\n\n\n\n

                                                          Predict volume density from coordinates only and emitted colour from full input to ensure multiview consistency<\/mark><\/strong><\/p>\n\n\n\n

                                                          \"\"<\/figure>\n\n\n\n

                                                          What does this represent? Do both coarse and fine nets have this architecture?<\/mark><\/strong><\/p>\n\n\n\n

                                                          Volume Rendering with Radiance Fields<\/h1>\n\n\n\n

                                                          Use classical volume rendering principles to render the colour of any ray passing through the scene using predicted local density and colour<\/mark><\/strong><\/p>\n\n\n\n

                                                          To render a novel view, we need to shade every pixel of the image: trace a ray through each pixel and estimate the colour of this ray.<\/p>\n\n\n\n

                                                          Volume rendering equation<\/h2>\n\n\n\n

                                                          Volume rendering equation gives the expected colour [math]C(r)[\/math] of a camera ray [math]r(t)=o+td[\/math] with near and far bounds [math]t_n[\/math] and [math]t_f[\/math]<\/strong><\/p>\n\n\n\n

                                                          [math] C(r) = \\int_{t_n}^{t_f} [\/math] [math] T(t) [\/math]<\/mark> [math] \\sigma(r(t)) [\/math]<\/mark> [math] c(r(t), d) [\/math]<\/mark> [math] dt [\/math]<\/p>\n\n\n\n

                                                          [math] T(t) = \\text{exp} ( – \\int_{t_n}^{t} \\sigma(r(s)) ds ) [\/math]<\/mark><\/strong><\/td>[math] \\sigma(r(t)) [\/math]<\/mark><\/strong><\/td>[math] c(r(t), d) [\/math]<\/mark><\/strong><\/td><\/tr>
                                                          Accumulated transmittance from [math]t_n[\/math] to [math]t[\/math]<\/mark><\/strong><\/td>Volume density at [math]r(t)[\/math]<\/mark><\/strong><\/td>Colour at [math]r(t)[\/math] when looking in the direction of [math]d[\/math]<\/mark><\/strong><\/td><\/tr>
                                                          “How much stuff have we encounted along the ray up until the current point?”<\/mark> \u2192 is a function of previous densities<\/mark><\/td> “How much stuff is there at the current point?”<\/mark><\/td>“What colour is the stuff at the current point?”<\/mark><\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n