The Paper That Turned 2D Photos Into Perfect 3D

[00:00:00]Welcome to 100 days of research papers.

[00:00:02]This is day 16 of 100. Today we are covering Nerf, representing scenes as neural radiance fields for view synthesis. Published in 2020 by researchers at UC Berkeley, Google Research, and UC San Diego, this paper revolutionized 3D computer graphic.

[00:00:22]Imagine taking just 20 or 30 photos of a statue from different angles with your smartphone. From just those 2D images, could a computer generate a perfect photorealistic 3D model that you can view from any angle? For decades, this was incredibly difficult. Traditional 3D reconstruction created messy wireframe meshes. They struggled with complex lighting, you know, reflections, and transparent objects like glass, water.

[00:00:53]Nerf introduced a wildly different approach. Instead of trying to build a physical 3D shape out of polygons, Nerf uses a neural network to learn the behavior of light in a specific space.

[00:01:06]You feed the network an XYZ coordinate plus the viewing angle. The network outputs a single color and the density of that specific microscopic point in space. It doesn't store a 3D model. The neural network itself becomes the scene.

[00:01:23]To render a new image from a camera angle that never existed in the original photos, Nerf shoots virtual rays of light through the scene. Ask the neural network what color each point should be and adds them up to create the final pixel. The results were magical. Nerf captured perfect reflections on shiny cars, the translucency of a leaf, and the complex geometry of a wire fence.

[00:01:53]Things that traditional 3D scanning completely failed at. Within a year of this paper an entire industry sprang around it. Today apps like Luma AI let you create Nerf on your phone in minutes. Google Maps uses Nerf technology for their immersive view to let you fly through digital replicas of restaurants and landmarks. Nerf taught AI how to understand 3D space by mastering the physics of light. This was day 16 of 100. See you tomorrow.