Generating Autonomous Vehicles Neural Reconstruction With Open-Source Physical AI Agent Skills

Added: 2026-06-03

[00:00:00]Hello, my name is Bruno and in this tutorial, I'm going to show you how to use the neural reconstruction launchable on NVIDIA DRIVE. Before jumping into DRIVE, let's quickly cover what Nurek is. In reality, Nurek takes real sensor data, either camera or lidar, and reconstructs it into photorealistic interactive 3D simulation. Today, it is widely used for physical AI development by NVIDIA and customers. Robotics teams use it with Isaac Sim, and their AV teams use it with Alpha Sim and Carla.

[00:00:26]The Nurek pipeline has a lot of moving parts. There's data ingestion, reconstruction, training, rendering, and artifact cleanup. That's exactly what the agent skills on this launchable are designed to handle. Instead of running each step manually, you just describe what you want and the agent sequences everything for you. We package it all into a DRIVE launchable, so you're up and running in one click. There's no need for a local setup or configurations.

[00:00:48]Let's take generating data for AV development as an example. When your fleet captures footage with cameras in one configuration, we are limited by the data captured in that single drive.

[00:00:57]Scheduling additional drives for data collection to gather different reconfigurations is expensive and slow.

[00:01:03]With Nurek, we take the recorded sensor data, reconstruct it into a 3D scene, and then can re-render the sensor rig from any position, field of view, or trajectory without going back out on the road. Today, NVIDIA runs workflow this workflow in production on millions of simulations per week to help validate AV model development.

[00:01:23]In this tutorial, we'll use Nemo flow as the agent interface, but the skills will work with any agent, so you can run them on your own structure as well.

[00:01:31]We're going to take an already reconstructed scene and ask the agent re-render it from a new sensor position.

[00:01:37]All right, let's get into it.

[00:01:39]>> [snorts] >> So, let's go to drive.nvidia.com, click launchables in the top navigation, and find the neural reconstruction launchable under physical AI. Click create agent. Once it's deployed, click open cloud.

[00:01:53]DRIVE spins up a fully preconfigured environment with all the agent skills, dependencies, and credentials already loaded. No setup is required. Once it's live, you'll land in the Nimo Cloud chat interface.

[00:02:06]All right, so let's try a prompt like render clip ID as if captured from a different sensor. Translate the front camera 0.5 m up and show me a comparison between the original and the new clip.

[00:02:18]You'll see that you'll see the agent calling various tools and running the full workflow end-to-end, giving you updates as it goes.

[00:02:24]This is Agentic AI. It answers your questions and breaks down the task into step-by-step pipeline.

[00:02:30]When it's done, you'll get a summary with the output videos inline and a link to web viewer. In the viewer, you can toggle between overlay and side-by-side mode to compare the original and the re-rendered clip.

[00:02:41]That's one use case, but there's a lot more you can do with these skills. A few other prompts worth trying, you could say something like get raw sensor data from Hugging Face and create reconstructed USDC scenes then show me the renderings of each sensor compared to the ground truth clips side-by-side.

[00:02:57]Or you could say harvest 3D assets from the physical AI dataset on Hugging Face and give me the USD files and orbit renderings of each asset.

[00:03:05]You can also scale the workflows with Osmo, such as reconstruct 500 clips with Osmo at scale in Azure.

[00:03:13]So, head on over to breadth.nvidia.com and try it out yourself. Or if you want to try the skills on your own agent and infrastructure, you can also grab them from GitHub at github.com/nvidia/skills.

[00:03:26]Both links are in the description below.

[00:03:28]Thank you. Bye.