Robotics' End Game: Nvidia's Jim Fan

[00:00:02]And up first, I'm delighted to introduce my friend Jim Fan. Uh Jim leads the embodied autonomous research uh group at NVIDIA, otherwise known as NVIDIA Robotics. Um I think that robot robots are just one of the most thrilling things that's going to happen. Uh a car basically is a big robot, but I'm excited for robots can go beep boop and lift things for us. And so Jim was Jim was a standout at last year's AIN, and we're delighted to have you back.

[00:00:28]>> Thanks [applause] everyone. Thanks.

[00:00:30]So, it was a summer day in 2016.

[00:00:33]Actually, right in this office that we're sitting in, [snorts] there's a guy in shiny leather jacket, you know, big biceps, hurling in this large metal tray. And on this large piece of metal, he wrote, "To Elon and the OpenAI team, to the future of computing and humanity, I present you the world's first DGX1."

[00:00:55]So, that was the first time I met Jensen. And as any good intern would do, I rush to get in line to sign my name on it. So, can you spot it? My name is here. And can you spot another? That's Andre right there. So, Andre, we're going to the computer history museum. I feel like a dinosaur.

[00:01:16]You know, back then, I had no clue what I was signing up for. And then, no one can describe what happened next better than Ilia himself. If you believe in deep learning, deep learning will believe in you. And oh boy, did deep learning believe in all of us big time.

[00:01:36]Three-step functions, six years. That's how all it took to bring us here today.

[00:01:42]The first take GP3 pre-training next token prediction is really about learning the rules of grammar, the shape of language. It's about simulating how thoughts and code and strings in general should unfold.

[00:01:57]2022 instruct GPT supervised fine-tuning align the simulation to do useful work or one reasoning using reinforcement learning to surpass imitation learning and finally auto research accelerating the whole loop beyond what's humanly possible. So, as Andre said, all the labs are getting to the final boss fight. So, for LMS, they're in the thick of the end game. And honestly, I'm very jealous. Look at how happy Andre was.

[00:02:30]Big smile on his on his face. The OM folks are having the party of their lifetime. They're speedrunning AGI on mythical creatures literally called mythos. So, why can't robotics get a piece of fun?

[00:02:47]So as any self-respecting scientist would do, I copy homework and I give it a new name. I call it a great parallel.

[00:02:56]So instead of simulating strings, can we simulate next physical world state and then we can align through action fine-tuning onto a thin slice of that simulation that matters for real robots.

[00:03:09]And we let reinforce learning carry the last mile.

[00:03:14]And that's it. the great paralo coping the LM success. If you can't beat them, join them. So, please join me in a new episode, robotics, the endgame. And sorry, I just couldn't resist. Nano banana is too good. Thanks, Dennis.

[00:03:31]So, how do we play the end game? It boils down to two things. Model strategy and data strategy. Let's look at a model first.

[00:03:39]The last three years were dominated by VAS or visual language action models and models like pi and Groot fall in this category. So we assume that the pre-training is done by a VM and we simply graph an action head on top of it. But really if you think about these models they are LVAS because the most amount of parameters are dedicated to language. So language is first first class citizen followed by vision and action. And by design VAS are great at encoding knowledge and nouns but not so much at physics and verbs. It's kind of head heavy in the wrong places.

[00:04:18]This is my favorite example from the original VA paper. Move the coke can to a picture of Taylor Swift. Yes, it has not seen Taylor Swift before. Yes, it's able to generalize. But this is not quite the pre-training ability that we're looking for. So what's the second pre-training paradigm? And I always thought that it would be something glorious.

[00:04:39]Unfortunate, it turns out that this is AI video slop that we call. You know, I can watch these um cats playing banjo on security cam all day. It's peak internet.

[00:04:51]But really, look at this. No one can take this seriously [laughter] until we realize that these video models are learning to simulate next world state internally. So these are some rods from V3. You can see that the models they pick up gravity, buoyancy, lighting, reflection, refraction all by themselves. None of this is coded in physics emerge by predicting the next blob of pixels at scale.

[00:05:21]And even visual planning emerges. Look at how VO solves these mazes. It solves them by running simulation forward in pixel space. And draw attention to the lower right corner here. This is my favorite example. Let's watch. And you blink if you miss. How V3 solves this one. [laughter] It's super smart. You know, V3 figures out that if you're not looking, geometry is optional.

[00:05:49]I call this physics slop.

[00:05:53]So how do we make these world models useful? Well, we do action fine-tuning.

[00:05:59]We align this superp position of all possible future states and [snorts] collapse that onto a thing size that matters for real robots.

[00:06:09]Introducing dream zero. is a new type of policy model that dreams a couple of seconds into the future and acts accordingly.

[00:06:19]And you know that motor actions there are highdimensional continuous signals.

[00:06:24]So that looks just like pixels. We can render it at the same time as we render the videos. So dream zero zero jointly decodes the next w states and next actions and as a result it's able to zeroot solve task and verbs that it has never seen in training and as a robot executes we can visualize what it's dreaming about and the correlation is very tight. If the video prediction works the action works if the video hallucinates the action fails. So once again vision and action are now first class citizens.

[00:07:00]And we have a lot of fun with Dream Zero. So we just wrote a robot around um in our lab and then type random things into the prompt box. And of course Dream Zero is not going to get all of these tasks 100% robust, but it's kind of like GPU 2. It's trying to get the shape of the motion correct in every case. So Dream Zero is our first step towards open-ended open vocabulary prompting for robotics.

[00:07:27]And we [snorts] call this new type of model world action models or wh.

[00:07:32]So let's all take a moment of silence for our dear friend VAS.

[00:07:37]They've served us well. Rest in peace.

[00:07:40]Long live world action models.

[00:07:43][clears throat] And next data strategy.

[00:07:46]This is Nvidia's chief scientist Bill Dali operating teley operation inside our lab. And given his salary, I think this is by far the most expensive teleop trajectory ever collected in our data set.

[00:08:01]The past three years have been dominated by teley operation. It's the golden era, right? VR headsets, extremely optimized latency for streaming and these complex rigs that look like medieval torture devices, you know, so much investment in industry, so much pain and suffering.

[00:08:22]And yet for teleyop it's upperbounded by 24 hours per robot per day. The fundamental physical limit. And actually who am I kidding? It's more like three hours per robot per day and only when the robot god is merciful because they throw tantrums all the time. So how can we do better? Well, how about this? You just wear the robot hand on your own hand.

[00:08:45]So this is called UMI or universal manipulation interface and it's a deceptively simple idea. You wear the robot actuator on your hand and directly collect the data as humans while getting the rest of the robot body out of the loop. Yet I would say UMI is perhaps one of the greatest papers ever written in robotics data and it spawned two unicorn startups. On left hand side is generalist improving this design. So you can wear the gripper here and on right hand side Sunday made these three-finger data gloves. So last year we took it one step further. We designed this exoskeleton that has a onetoone mapping with five finger dextrous robot hands and we call it dex ooi. Let's look at it in action. On the left the human directly collecting data always the fastest. on the right. Look at how difficult teleop is right the human operator here one of our most skilled PhDs he has to align very carefully right and then it's super slow also the success rate is very low as well and in the middle you just wear these exoskeleton and you collect data directly and we train a robot policy on this data so here what you see is a fully autonomous row out of a policy that's trained on zero t operation data So, we're able to break the curse of 24 hours per robot per day and see how happy these robots are because they no longer need to be in the loop for data collection. So, is this the answer? Have we solved scaling for robotics?

[00:10:21]Anyone driving Tesla or Whimo here?

[00:10:23]Anyone? Right. You know, when you're driving, you're actually contributing to the biggest physical data flywheel.

[00:10:33]And the beauty is you don't even feel it during FSD because the data upload is an ambient process. Yet wearing these UMI or data wearables, it's still cumbersome, right? It's intrusive. It's not as seamless as just driving to work.

[00:10:48]So we need an FSD equivalent.

[00:10:53]The data collection needs to get out of the way, fade into the background so we can capture the full glory of human dexterity across all walks of lives, across all labors of economic value.

[00:11:06]So we are going all in on human egocentric videos that come with these detailed annotations like hand position tracking and dense language annotations.

[00:11:17]introducing ego scale where 99.9% of the training that goes into this is based on human egocentric videos and the result is an end to end policy that maps directly from the camera pixels here to 22 degrees of freedom high dexterity robot hands which you see here is fully autonomous.

[00:11:39]We pre-train eagle scale on 21k hours of in the wild egocentric human data with zero robot data whatsoever. And during pre-training we predict these hand joints and wrist poses. Then in action fine-tuning we collect only 50 hours of high precision moap data gloves and four hours of teleyop. That's four hours of teleyop less than 0.1% of our training mix.

[00:12:09]And with this ego scale is able to generalize to these very dextrous tasks like sorting card or manipulating syringe right over transferring the liquid. You know someday we might have robot nurses at home. Might as well try this. And for these tasks it takes only one shot demonstration at test time to learn different shirt folding strategies.

[00:12:33]And perhaps the most fascinating finding from the paper is that we discovered this neuroscaling law for dexterity.

[00:12:42]It's a very clean relationship between the amount of hours we put into pre-training and the optimal validation loss. In fact, it's a clean log linear mathematical equation six years after the original neuroscaling law for language models.

[00:12:59]So if we put all of these data strategies on this chart, X-axis is alignment to the robot hardware. Y-axis is scalability. This is what it looks like. Teleyop the least scalable data wearables. You can go up to hundreds of thousands of hours. And egocentric video if we're able to spin the FSD flywheel easily 10 million hours in the next year or so. And if we draw a line here, everything to the left of this line is a new paradigm sensorized human data. So let me make a few predictions. In the next year or two, we'll see teleyop dropping and dropping to almost negligible amount. And then there will be an ensemble of data wearables custom designed for different hardware and use cases. And finally, the main diet for robotics will be egocentric videos.

[00:13:49]So, a moment of silence for our dear friend Tal. You have served us well.

[00:13:54]Rest in peace. Long live sensorized human data.

[00:13:59]Are we done with the data strategy yet?

[00:14:01]Did you notice I put two rings on data strategy? What's the outer ring here?

[00:14:06]All the OM frontier labs have spent significant budget now on acquiring millions of coding environments to do reinforcement learning. So, robotics is the same. were in urgent need to scale up environments.

[00:14:20]And of course, you can always do reinforcement learning directly on the real robot. So in our lab, we use RL to push certain tasks to almost 100% success rate so you can do these continuous execution for hours on end.

[00:14:35]You know, it's kind of therapeutic to see these robots assembling GPUs just by themselves or as a wise man will say, "Good boy, this task has been approved by my boss."

[00:14:46]Yet we can't [snorts] get to one million environments because that will require one million robots if you do it the previous way. So we need a better way here. Let's say you take an iPhone picture and you can pass it through this 3D wall scan pipeline to extract all the objects and then automatically synthesize them again inside a classical physics simulator. So all these objects are actually interactive after the scan and then you can augment this infinitely in simulation with variations that we call digital cousins.

[00:15:21]So now iPhone basically become a pocket world scanner. In this process that we call real to sim to real and in this way we have a scalable way to port the physical world into the digital world.

[00:15:35]But still this method relies on a classical graphics engine. Can we do better? Introducing Dream Dojo.

[00:15:44]So, it's always been on video world model and turning them into full-fledged neural simulators. Dream Dojo takes as input these continuous action signals and outputs the next RGB frames as well as sensor states in real time. Not a single pixel you see here is real. And Dream Dojo is able to capture and learn the mechanics of different robots through a purely datadriven approach.

[00:16:10]There's no physics equation, no graphics engine involved in this process.

[00:16:16]So the new post- training paradigm for robotics is a massively parallel RO system that runs on a few real robot stations on a bunch of graphics cores running world scans and heavy inference compute running world models.

[00:16:32]Or as this equation goes, compute now equals environment now equals data. Or as a wise man would say, the more you buy, the more you save. And this message has been approved by my boss.

[00:16:47]So that's it. Putting it together, the great parallel that robotics will follow. And it's happening as we speak.

[00:16:54]And we're looking at the beginning of the endgame.

[00:16:59]You guys play the video game Civilization.

[00:17:03]Still my favorite. I like to think of my research as unlocking game achievements on this civilizational technology tree.

[00:17:12]>> [clears throat] >> And there are three more achievements to unlock for robotics and then we're done.

[00:17:17]I can retire and I can't wait for that.

[00:17:21]The first is passing the physical touring test. Across a wide range of activities, you cannot tell the difference between a human doing the task or a robot doing it. Maybe not drunk humans, but you know, physical touring test is about unit energy in and unit labor out. And just by judging at the sexy pose of this robot, I think the work is cut out for us. So maybe it's two to three years away.

[00:17:51]And next physical API, you have a whole fleet of robots and they can be configured just like any other software using APIs and command lines orchestrated someday by Opus 9.0. Z and if we have this physical API we'll be able to realize lysot factories those are essentially printers of atoms they take as input design in markdown files and then output fully assembled products completely autonomous or these wet labs that automate scientific discoveries in chemistry biology and medicine and the final stop physical auto research when robots start to design, improve, and build the next iteration of themselves far beyond what's humanly possible.

[00:18:40]So, you might ask, is this too science fiction? Like, are we going to see this in our lifetime?

[00:18:47]Well, it took the AI community 14 years to go from the first forward pass of Alexet in 2012, a model that barely recognized cat versus dog to AI ascent today 2026. Well, we talk about agentic auto research and let's just add another 14 years. How about that?

[00:19:08]2026 is right in the middle of 2012 and 2040. And technology does not advance linearly. It advances exponentially.

[00:19:19]So, [snorts] I can say with 95% certainty that we'll get to the end of the endgame, the end of the technology tree by 2040.

[00:19:30]And we'll still be all we'll still be on.

[00:19:34]If you believe in robotics, robotics will believe in you. And to all of us here sitting here, I think our generation was born too late to explore the earth and too early to explore the stars. But we are born just in time to solve robotics.

[00:19:54][applause] >> [applause]