VLK: Learning Humanoid Loco-Manipulation from Synthetic Interactions in Reconstructed Scenes

[00:00:00]Welcome. We introduce VLK, a system that teaches a humanoid robot to see, listen, and act in the real world. Here is what it can do. Our first contribution is a pipeline that generates training data automatically inside reconstructed real world scenes. Step one, scene reconstruction. We scan a real room with an iPhone and reconstruct it using 3D gshian splatting. The result is a metric scale photorealistic copy of the environment. Step two, waypoint generation. Using the scene geometry, we sample task relevant waypoints. Feasible spots for the robot to walk to, pick from, or place onto. Step three, robot motion synthesis. We synthesize Unitry G1 whole body motions that follow these way points. For both navigation and box interaction using a conditional diffusion model. Step four, egocentric rendering. We replay the synthesized motions inside the 3DGS scene and render what the robot would see through its head camera. We also apply visual domain randomization while rendering, varying lighting, color saturation, and camera parameters. On the left, no randomization. On the right, with it, this helps the policy transfer from rendered images to real ones. Putting it all together, this pipeline generates 48,000 paired trajectories per scene fully automatically. Now, let's look at what those trajectories look like.

[00:01:49]With paired data, we train a vision language kinematics policy. It maps what the robot sees and what it's told to do into the motion it should produce. The policy takes three inputs. The egocentric image, the language instruction, and the robot's current kinematic state, including wrist contact. During training, we use ground truth contact from the data set. At deployment, the contact is predicted auto reggressively from the model's previous output. We initialize from a pre-trained PI0.5 vision language action model and fine-tune it on our synthetic data. The policy predicts a 1second future whole body trajectory at 30 hertz joint angles, root pose, and wrist contact labels. A separate whole body tracker trained in simulation converts these kinematic targets into joint level commands the physical robot can execute.

[00:02:36]The whole pipeline runs in real time with 31 millisecond inference latency.

[00:02:40]That completes the policy. After training, we deploy our model in the simulation and real world.

[00:03:14]finally we show that the VLK policy trained on purely synthetic data can be executed in the real world zeros task one navigation given a language instruction the robot walks to a target object Even with layouts it has never seen during data generation.

[00:03:45]On the left, the robot starts from a random pose. On the right, the language instruction was never seen during training. Both work. Zero shot.

[00:03:56]Task two, box lifting. The robot picks up boxes of three different sizes from the floor, small, medium, and large, and puts them back.

[00:04:14]Task three, multi-stage tasks. By streaming language instructions in real time, we chain atomic skills, walk, pick, carry, place into longer behaviors.

[00:04:40]Task four, robustness to disturbance. On the left, we change the scene layout while the robot is acting. On the right, the robot operates under flickering disco light.

[00:05:26]Task five, long horizon tasks. By chaining navigation and pick and place with real-time language, the system completes multi-minute behaviors.

[00:06:19]VLK shows that synthetic interactions in reconstructed scenes can supervise real humanoid loco manipulation. Thanks for watching.