Ep: From Capable Controllers to Deployable Humanoid Systems

[00:00:00]Hey everyone, welcome to another episode of Robo Papers. I'm very glad today to have Litu or Gary Young um from Caltech.

[00:00:07]He's one of the co-authors of this paper called Hand Offoid agentic task space whole body control via distilled complimentary teachers. Uh so thank you so much for making that time Gary. Um perhaps as a start you can introduce yourself a little bit and then we can dive into the paper.

[00:00:22]>> Right. So my name is Gary. Uh as Michael introduced I am from Caltech working with Professor Era Ames. Um topic of my talk today is C from cable controllers to deployable humanoid system. So I personally work in the humanoid space and you know we see a lot of these very cool demos nowadays right especially the most current demo from Sakon where they did a 7 minute autonomous long form long horizon task execution video that was very cool but I want to understand what can we how can we move these things further right how do we make them safer how do we go from very capable humanoid dancing or humanoid task operation to actually deployable humanoid systems in the Right. So yeah, fourth year PhD student. I was at UC Berkeley and back then worked mostly on robot experiment deployment which is why I'm more of a I guess engineering minded PhD in my cohort and uh experienced robot doctor. My peak rate in one day was I uh my labmates were running robot running experiments and we had to fix it three times in a day. I think my personal record is doing these kind of like replacements in honor an hour just last week. So as we know humanoid capability frontier is moving very fast right like we have very nice results such as the uh boss dynamics results and also the Nvidia solid controller and the frontier has shifted from can aerobos move can human cumoid move agile in the world you can operate as a system and as we know learning controllers making behaviors more expressive and less handd designed such as can be seen here. So in my mind like demos like these show capability but then deployment in the real world reveals the assumptions capability demos are the right starting point right which we demonstrate what a human robot can do in the world how the motors can react can it be faster quicker higher um but then deployment asks what the demos are assuming what commands do we choose so that is intuitive for the human planner or for any high level task planning system can the robot react to the environment such that It doesn't do things like this where it just kicks a kid and how would controller handle postal failures at refresh level right preferably not just estop and then crash on the spot. So the challenge here then shifts from producing motion to building a systems around it which contains interfacing having the robot understand the context safety and also how do we inject all of these into a learning part. So real world deployment means layer defenses right like things will never be perfect no matter how robust we make it failure often happen when assumptions line up across the stack in what we call the cheeseful model in safety. So commands that controllers will not follow safety constraints and misses context and learn policy exploits a wrong shortcut and all these line up we arrive at accident and closing these gaps require layers that connect intent to emotion context is safety and safety back into learning. So we can only patch up these holes as much as we can, right? So we can never make the assumption that one layer is always foolproof. It is our job to make sure that across all layers we have enough patches such that holes really line up and we can't make the assumption that holes never line up too, right? Have to be always vigilant against these kind of failures. So which leads today to of course the handoff paper and also some other papers that I did in the past few years which is my journey into the three layers for deployable human system which first of all is the agent interfaces where instead of using something in latent space or just producing whole body motion we provide a very clear command interface such that it is easy to guard against and also after passing this interface command down into the directory level. We've explored ways to generate good trajectories based around semantic understanding of the environment and also taking into assumptions of the learn controllers capabilities. And lastly, we try to find ways to inject this kind of safety notions into the actual robot policy learning itself to provide controllers that understand the notion of safety inherently. So the first thing is of course the most important topic today which is handoff cable agentic task based whole body control that is still complimentary teachers. So the idea of this project when I first started was to bridging was to bridge high level intent and whole body motion because when I started doing this uh in the start of this year we have seen papers from Nvidia right like the sonic paper and also their demo from last coral I think where they said okay this robot is able to go to the vending machine get me a cup of coffee. Um so I was thinking how can we find a very good architecture for that right like not say switch between controllers or have to generate a whole body motion tracking policy uh whole body motion tracking motion for that to happen. So I thought that high level autonomy needs commands that can easily synthesize and be interpretable. So something that we can understand as well as the machine can understand and one way to do this if we follow say a vidia's route is we have very dense hob references um there is expressive but they are very all core for planners VM agents and task policies right like for example in this in here in this demo here where we want to pick up a bottle a whole body reference has to generate the whole reference over there and has to generate every frame right so it's for say is 10 generates like 500 frames for 50 Hz. So what we do is that we try to narrow the planner and controller boundary without giving a whole body behavior. Right? For example, with our controller, we can command to walk, to squat, and then also to use his hands to pick up the bottle, which we uh essentially extracted this very nice formulation into the uh controller interface, which is the velocities, the hand positions and and plenty orientation later and also the uh pelvis height. So, in order to do this, we also try to make it a unified policy, right?

[00:06:31]like um because a lot of these papers even the flexion demo I think what they did is that they use different policies for different parts of the set which is completely fine right but I wanted to try to bring this a step further where to see if I can make everything unify a policy. Um to do this I initially started actually just trying to see okay we have this very nice whole body tracking controllers like sonic like twist how can we just distill it down into a policy that takes our defined 10 uh 10 dimension no command right like but then I found it's really hard to do it the reason being um whole body tracking control policies needs a very good conditioning signal for that and a very good one it turns out is actually the whole body reference you pass into Right? Like why twist 2 is able to say walk move its armors about jump jump a bit I guess and also dance a bit is because it has this very uh strong prior a very strong observation that's passed into it that's a whole body reference um if I take that out which I did uh I found that it's the learning really collapses so my as path to that is I introduce these complimentary teachers right there's a whole body teacher There's the local motion teacher and there's the recovery teacher. So for the whole body teacher, it is used to handle most of the uh whole body motion that's needed while the robot is doing some local manipulation task, right? For example, reaching out, squatting, using both of his arm to hold something so and so on. So for the local motion teacher it is a very simple taskbased local motion policy since a common issue with these kind of mimic human motion policies is that they don't track those velocity domains really well and what I do is I went back to the basic local motion policies right so like um I guess even going back to doing newspaper on Cassie following that stat the local motion policy there >> sorry can I can I ask can you can you elaborate on a bit. Why wouldn't they be able to track low velocity skills very well?

[00:08:39]>> I think it's just more just because um if you go the mimic route, right? Like both AM and whole body tracking like deep mimic sorry beyond mimic. The problem there is the data you usually we usually use to train these don't include low velocity human walking like when we walk we don't like uh shuffle for 02 seconds 2 meters per second right like right the data is just all there um I'm pretty sure if we collect the data it will work right like we could have that data coverage >> but we don't so >> so it's just a data distribution mismatch issue not like a fundamental >> distribution nothing yes I also tried on to Okay, so the interesting part here is I also try to generate these kind of low velocity domain data with IK, right? And I try to say, okay, I have IK, I have human data. I'll just add these together and train the local policy with that.

[00:09:31]Also did not work really well because turns out there's a very big style difference between the human motion and the IK motion. So it's really learned a human policy where it can also easily switch between these two because these are very disparent. Um yeah and I think one of yuka's PhDs also found something similar with VAS and had a very nice paper on that um about like the different style data and how do you manage the data coverage problem. Yeah.

[00:10:02]Okay. So going back to this in the end I used three teachers one for handling the local manipulation one for handling the local motion while taking the arm and upper body motions into account and the the last one handling recovery. And a very nice thing here is that because these can be divided into different domains, right? I can use velocity gated blending and also just conditional gated blending for the recovery such that the gradients of these don't really conflict with each other, right? Like so I don't really So, so I can blend these skills together nicely. So, so for example, when it's on the ground, it never tries to learn to walk, right? When it's on the ground, just tries to learn to recover. Yeah. Um I use the MOE architecture for the student increase capacity. Um I it might not be needed. I just didn't really have a lot of time to go into how do I scale these policies like for example solid. Um this policy in the end is I think uh five to times smaller than solid's parameter count. So it's is also very efficient. Um and the good thing about this is that keeps one interface for covering multiple motion regimes. Uh going back to the data issue, one thing I did is I made the data more dynamically feasible by filtering them with a contra barrier function using the uh center of pressure as the condition. Right? So for example here you can see without the filter when you're tracking a reference motion the robot just goes backwards because the center of pressure is towards the back of the whole polygon just like on the edge and slightly slightly over the edge. Well, if you filter it, the robot can just stay stable. And in the end, it can do all these things, right? Like walk, track, a nice velocity command, squat, bend down, and all the motions that you would think of while humanoid is commonly used for. And also the last thing is like I can also, you know, just get back up. I didn't really try this on a real hardware because we only have one G1 and I strap like a four on the back of it. So, uh, it's it it comes some cost. Yeah. So after having arrived this controller, the next thing I saw is okay, how should I make it do the things I want, right? And that's what led to this agent controller interface part. So this is also built as example to say with this controller I can essentially give it any command like I can give it a lot of commands, right? And I can also just slap on you know this very easy controller and will also provide very nice results. um people are welcome to try their own try their own planners at it right like doesn't necessarily have to be in this formulation maybe you can hook up VA to it maybe you can hook up something else to it like this is just a very simple instantiation of that philosophy but here what we do is that the language instruction will decompose in atomic tasks so for example if I tell it to pick up the Pringles chip bottle and put it the left of the cheer box first we use a task composition node to decompose into very automatic task test.

[00:13:02]For example, I find and navigate to the Pringles bottle, reach and grasp it, find navigate to the cheeser box and place it to the left of cheer box. So here a very interesting thing is I actually did not go sort of this bindy to the language model. So what I did is I remember reading I think the cloud code news and one part saying they use like redex for some initial filtering.

[00:13:24]So I took some inspiration from that and as well use redex for my initial initial filtering. So if there's any connecting words like and then whatever I just slice it by that and if that end result is still not autoomic that's why I pass it to a language model and prompt it to say okay I need to decipher this into a very autoomic tasks right the reason for doing this is I initially wanted to run everything locally so that's also part of reason I strapped it for to it and running it locally means that inferencing is not as fast as running let's say with chatgbt with cloud right it's through the APIs. So how do you do some speed up with this? And after it has been decomposed into these kind of tasks, we can then call different tools I guess in the agent language to instantiate these tasks, right? Like for example, find the find tool can be translated to using the RGBD image and the BLM to produce pelvis frame way points. So for example the Pringles box given the RGB image given the point cloud I can find where the Pringle box is and also similarly for the cheet right like and to the left to the right since the VLM also has some notion of these kind of spatial relationships in the RGB space right like we can use it to handle these kind of things and after having these way points the scope selector selects root height and wrist targets and also velocity motions to velocity commands to move the robot to where it's supposed to be. And lastly, we use the handoff controller to track these commands and to complete the motions as can be seen here.

[00:15:01]>> Um, hey Gary. Um, sorry, I have a couple of questions. So, yeah, >> I'm I'm quite curious like uh when you're doing the VRM, I mean this this decomposing into atomic tasks, right?

[00:15:11]Like for example, here it says find and navigate. Like in your experience when you're prompting the VRM to do it like like does it do a really good job? Like for example, find and navigate could arguably be broken into like two atomic tasks. I mean, it always strikes me that there's almost like a qualitative nature to what is defined as atomic. Like find and navigate is like find first and then you know.

[00:15:35]>> Yeah. Yeah.

[00:15:36]>> Or find while navigating.

[00:15:38]>> Personally, I I designed this to be a like bottom up approach where I design the tools first, right? And then define based on these tools what the atomic task is. So for example the tools I have for this uh auto for this like framework is given a waypoint navigate somewhere right given a waypoint um move your whole body to make sure your like hand arrives at waypoint so here I can classify find and navigate to be a one atomic task because find is essentially in the RGB frame generate that waypoint and then navigate just walk there right so that's why like here I can classify it as a task similarly with uh like pick up, reach and grasp and also like place.

[00:16:20]Yeah. So, it's all defined I think on the tools you build, right? If you think that um curing the VLM for the waypoint and navigating is like two separate tools, then yes, like there should be two separate tasks. Yeah, I see. Um and then I guess my next question is that you know the skill selector like setting the root height and and wrist targets.

[00:16:40]Is this a very trivial thing to do? I mean once it has a you know this pelvis frame way point um >> yeah um it's just yes yes it's it's not like a very sophisticated thing right like given these way points in the uh robot's pelvis frame is easy to set these things right like for example if the because we know the capabilities of the controller right like if the item we want to pick is below the I guess lowest reach of the robot we need to also kneel now, right? If it's not, we can just move the hand to the pelvis target.

[00:17:14]>> Mhm. Got it. Okay. I like that.

[00:17:17]>> Oh, sorry. You You keep going with your questions.

[00:17:20]>> No, no, no. I I I think that's a funny.

[00:17:22]Um Yeah. Sorry, Chris.

[00:17:24]>> No, I was I was just wondering. So, I really like this predicting the way the way points, but do you how well do you think this would work? Like this is all these tasks that this sort of seems to work mostly just in front of the robot.

[00:17:34]Do you like if if your task involved like moving around a whole kitchen or so, how what would you need to do to extend it to that kind of setting like a full full kind of room?

[00:17:42]>> Yeah. So, one big issue with this kind of robot is that we don't have an actuated neck, right? So, if want to find something, I have to turn the whole robot. I do have like a um find and search task in there, right? Like where if I can see it, just turn it by like 90 degree quadrants to search for the item until I find it. um with as you can imagine with a neck or like with some with more like um pelvis y movement it will be a lot better right like they just turn the upper body just turn the head to look around for the item and it clearly self-extending this and I think has been done in like dimension west and also some other people's like open source software is that um how do you incorporate memory into this kind of facial agentic systems, right?

[00:18:34]Like as you go through a scene, how do you remember what's in the scene and like where everything is in the scene, right? Like goes back to people doing scene graph research and people do who who build like 3D scene graphs of the environment. Yeah. So like to I guess answer Chris's question of if you're in a kitchen, you have to like uh complete a very long tail task, right? Like for example, it has incorporates moving around and picking up different things, composing different things. Memory is I think very essential there.

[00:19:06]>> Yeah, makes sense.

[00:19:06]>> Sorry, I think I'm muted.

[00:19:08]>> Yeah. So, so in that case, so you would probably to like add in some kind of memory or something like this. No, it makes sense.

[00:19:15]>> Yeah, exactly.

[00:19:16]>> Yeah. Which is why do you want to make this work? Um it's more I am strapped with you know humanoids and deep [snorts] >> you understand the PhD type right so >> of course yeah >> right so here we briefly demo the diverse natural language driven tasks that we made it do right for example pick and place transport nearing and picking up something handed over to a to a person task continuation of the fall recovery only a sim sorry about that and a bit of a teleop So, you know, after we took all the other videos, right, like 3 days before the deadline, I'm like, you know, let's just do something fun. So, we took it outside and we did some telloff experiments where I just try to hit a batminton ball, which turned out to be fine. So, yeah, all in all, it ended up being good. So, sorry. Um, so I want to ask, so let's say the top right, right, where you have this picking up the blue box. So I mean concretely what's doing is even though it's just one like very broad thing it first program into many sub atomic tasks >> um and then >> right so for example for this one it's for this one it's like uh find a blue box right and then pick it up with both hands >> and turn around and hand it over is like turn around is one like you can also provide it with explicit like positional movement commands right and then hand it over is one. Yeah. So for example, this one there's four commands, right? Like find a box, pick it up with both hands, turn around, hand it over.

[00:20:49]>> Yeah. But I suppose the hand it over the last task mean it's uh it needs to know that that I'm looking for a person and then >> Yeah. Like for example, the last one the last one like I essentially said, okay, just just find a person and walk to walk to her and hand it over. Like I mean if there's multiple person in the scene, then yes, it will just find whichever person you found. Yeah.

[00:21:09]>> But even if sorry, just on this class as well, right? I mean so the the VRM is just creating the way uh I mean the the way point >> yes >> so and then you use the controller to convert into the sorry I mean the 10D and then convert into the the whole like 29 degree of freedom thing right >> yes so >> it doesn't really take into account force and all that right is sorry you mean force as in the weight of the box and stuff like that >> yeah because like when you you're doing visually, right? I presume, right? I mean, you it's just a point in a thing and then you you move your whole body towards that thing. So, it's I imagine it's very good for squatting down and probably quite easy if it's just like power grabs or something. But in this case, I'm quite curious cuz it's you're coordinating both arms and then but you're current carrying something that presumably have some weight even maybe it still has some weight, right? So, I wonder how um yeah, how do you compensate for literally the gravity of the the box? Okay, so just for full disclosure, this is more to like this demo here is more to dis to show that it also has like these kind of banging capabilities because all the other demos we did were single hand. Um, so the box really weighs nothing, right? It's just a cardboard box.

[00:22:24]>> But I guess the way I do it is that I squeeze in both hands, right? So like instead of saying, okay, I place my hand lightly where the box side is, right? I actually command it to go in a bit more to go into the box. But you know in a physics world you can't really squeeze into the box and by doing that you induce a force on the box to pick it up.

[00:22:46]Um one can imagine that you can probably do what uh professor Agro did at MIT. So like try to use the current readings from the motors to deduce how much force is needed for the box.

[00:22:59]How much force is need is exerted on the arms by the box. Um, but I do not do that here. Yeah, >> that's still very cool.

[00:23:07]>> And just to point out, my friend told me that he tried off Sonic with these kind of like box pickup. Um, Sonic can produce a motion but can't actually pick up the box because precisely because it's not like squeezing in, right? It is like just placing a hand to the side of the box like as a human. Yes. Because you know like you're exerting force sideways into the box, right? You can pick it up. But like if you're just motion imitating that, you can't really do it because for a robot just place his hands to the side of the box and it's never se squeezes in.

[00:23:35]>> Yeah. But I I would say I mean for example the the other one here where it actually squats down to pick up the the the the fingle. I think that one is Yeah. I mean it really showcase your your 10D controller. Um it is also very intuitively for maybe a human to think about like you know okay turn left turn right you kind of squat down a little bit and then you once you roughly there then you you know reach out on your arm while you're squatting down this yeah >> so yeah um have you seen other like controller that also like basically compress into just this 10D kind of uh space >> um that is very good question because as you know like in research now we like there's very hot topics going into human right and I I think for humanoid VAS there's really two methodologies one is like let's just generate the whole 29 of like motion directly right like given data collected like either use a humanoid um or like actually as a skeleton or like a hybrid of these we can actually back compute like back using back comput using IK the 29 do like joint positions of the humanoid and I just learn these directly through the humanoid or could do something like I think going back to media again like Sonic and Groot. So what they did if I remember correctly is that you first generate a latent representation and the solid decodes a latent representation into a like tracking controller right so it's kind of like what we're doing here but the thing there is that that latent thing in the middle the latent command is not desirable by humans right does this >> exactly >> less interpretable yeah >> I haven't really seen a lot of papers that does this in a more learning based sense I think Albert is uh Karen Leo's PhD at Stanford. He did a lot of like model based stuff uh on a mobile manipulator. Um it was very impressive.

[00:25:30]They can actually just use a model system to do all of this. Like I think he did a supermarket low like shelf loading and like shelf reorganizing and taking things out of the shelf with like a mobile purely model based. So that is similar to what we do here. I actually took some inspiration from the works that he has. like to use a more model based planner but then to the thing I did here is that I have a whole body learning based controller to actually have these uh more model based planning systems.

[00:26:01]>> Yeah, I mean I I I I like this tend a lot because it's it's very interpretable to a normal human like I roughly know what's going on. I can imagine almost like do the game controller thing or then once it's roughly there then I switch to the okay then I take it out.

[00:26:14]It's but it's roughly that's why I think the telly op thing it's it's translate very naturally you can just go so quickly because >> yeah so like >> yeah so the good thing about this is that you can hook up a lot of system to this right like for example the thing I essentially call it up in a day because >> uh the only thing I need to do is like hook up the hand controllers like positions to the robot positions through some like simple transforms like my arm is longer than the robot's arm right like and hook up the velocity say to the head to the like uh head VR unit velocity and that's it right like don't have to do anything special for encoding the commands in some data space or do some like very heavy IK stuff yeah >> yeah do do you have more video on the teleop actually do you try like for example instead of just standing here do you move it around um and then do some squatting as well >> I so so I did hook up the walking policy to like the joysticks fix the controller actually. Yeah. Okay. Let me see if I can pull up a video. Um, >> yeah.

[00:27:20]>> No, no. I I I should have put a slides, but yeah, like >> so basically >> Professor Yo was like terrifying on the ground while he was like teleoping it like you know, walking up to him like holding his holding the robot's arms out. But yeah, I guess I didn't put the video in the slide. Sorry about that.

[00:27:37]>> Okay, cool.

[00:27:38]>> Yeah. Yeah, I think it's [snorts] I think I can't find it here. Sorry. Yeah.

[00:27:42]No, no worries. No worries. Yeah. Um, how I'm also curious like how does this system treat like recovery at all? Like not not the kind of recovery that like like you fall down, you get up, but more like okay um maybe I'm I'm supposed to execute this particular uh subtask but then or atomic action but then actually in reality you couldn't complete it.

[00:28:07]Like for example, you you squat down, but you are your when you try to reach out your arm, you it's just a bit too far. So you actually need to redo the whole thing. It's um >> Yeah. So uh I do have that in the controller, right? So for example, because this gripper is a like not a servo driven gripper like it's actually like a um motor driven gripper. So I can kind of like know from the torque response and from the current response and just from like if it's closed fully, right? To see to know if I actually grasp something or not. If I did not grab something, I do have the logic in there to like reample and retry. But not resample, sorry, re relocate and retry.

[00:28:45]Yeah. Yes. But um I guess I just don't never show it in the videos.

[00:28:49]>> Yeah. Because I'm I'm wondering like uh it's especially like this VRM. I I always wonder like what is is in native understanding of visual space like cuz right now you're poking up like uh for example that that u demo you have here where you walk towards the the shelf and you squat down and then you reach out to graas the object right what if in the third step it didn't squat down in a place that is near enough that the arms is able to reach you know um I wonder if it's know what is the right way point to put for the third atomic task or is a problem at all. Yeah. Mhm.

[00:29:26]>> So currently the logic I had it do is that where it has a way point right it navigates to the closest free space. So for example if the item is on the table right like I navigate to the closest free space on the ground for the humanoid to stand. If it can really if if even then it cannot reach right that means the thing is just out of >> reach which I think it's not the uh like best way of handling it. The best will be like taking into account the humanoid reach, right? Like just the put the humanoid in the optimal place as you said to reach for the item. Uh for now I just said to be the most conservative, right? Just to move as close to the item as possible.

[00:30:06]>> Got it. I see. Okay. I mean Yeah. I mean that's a very reasonable thing to assume you might that's probably come majority of the cases.

[00:30:13]>> Um if you can't reach you probably have to use some tool to reach the thing.

[00:30:17]>> Yeah.

[00:30:17]>> But you know that's another paper I already >> Yeah. That's that's more of a log like task planning thing which >> I uh >> yeah which which is like as you said a future work but not really planned yet.

[00:30:31]>> Okay.

[00:30:32]>> Another thing that I I really like and I know you didn't do this on a real robot for obvious reasons but is the uh the falling down example where it gets back up. Like that's something that we just don't see a lot with humanoid robots. Do you think like uh because I don't really have a question there other than like do you think there's a world where where you could try that or why like could you try it to try to make it avoid the Jetson on its back or something or in the real world?

[00:30:54]>> Yeah. I mean like people have been doing a full recovery research for I guess at least >> Yeah. But not workflow, right? Like that's the Yeah. Go ahead, sir.

[00:31:03]>> Yeah. So, so um this for recovery is actually like it's actually a the t-shirt itself is a open source ambass baseline that I used the original orders I think it's called AMU or something like the MJ lab like open source GitHub repo. Um the original orders they have some videos on social media where they actually try to get a robot onto the ground having it recovered. Um but the reason they can do that is he I think he actually interns at Mitry. So he has like >> Yeah.

[00:31:43]>> like 10 robots to work with. I only have one so I can't really risk the robot trying it. Yeah.

[00:31:49]>> Well, but as you said, right? Like there's there's more there's like some ways of doing this, right? Like for example, if you are in F4, like use your arms to brace yourself against the fall, right? Like for example, maybe hold your arm to your back to protect the dress or to hold your arm to the front to like protect your like camera or like your head, right? Like stuff like this.

[00:32:08]>> I was thinking protect your fingers. But yeah, it makes sense. Like the hand hands on these things are expensive and break easily. But yeah. Okay. No, it makes sense. That's great.

[00:32:17]>> Yes. I guess that's the other thing as you said. Um like if we end up hooking up like Desert's hands to it, right?

[00:32:23]like one Dexter's hands is the cost of the whole robot. So yeah, as you said, we probably don't want to use hands to like Dexter's hands to actually help her recover to like to embrace a fall, right? Like that would be not good.

[00:32:36]Okay, so that's the first part of my talk on aic planning systems and also the agent oriented controller. Um, okay.

[00:32:46]So given we have a very nice aentic interface, right? So the next thing is passing it down to the mid level like uh trajectory planner and how do we handle safety and deployment there. So here we have two works. So one focused on learn knowing about the limitations of the lower level tracking compar focus on how do we use semantics at this layer to enforce things like safety and social compliance. So the first work is actually something I did a while ago called shield which is a humanoid CBS inspectation on learn dynamics. Um reason behind this is that we have very spective controllers right even back then you can train off a RL controller in like a day and then deploy it onto the G1. Now you can train it in like a few hours on the G1. Um but the problem is they are not safe due to dynamics inaccuracies right like you can tell it to track your 0.5 m/s command it will you know track 045 point 46 the average might be 0.5 but the instantaneous one might be not right like so it will inevitably do something that you don't expect so what we did is we came up with this method called shield to try to bridge the gap between the users expectations and the policies behaviors.

[00:34:07]Okay. Yeah. So how we did it is we uh first use a genative model to learn the tracking controllers difference to the user expectations right. So like given this continuous sign system we first linearize it into this form into this form and then we run a data collection in real world to essentially say quantify the uh error and the uh variance of that error prediction by learning a generative model and with this tracking filter we can then say given the user's command this is how much I need to modify to actually make the underlying controller track the user's command and then We can pass this all through a safety filter with that co with that like various I mentioned earlier so that essentially we have a safety filter that knows how sure that the general model is about the correction mate. So if it's not sure right we have to be more conservative if it is very sure about the correction mate that we can afford to be a little less conservative but we still don't want to violate the underlying conditions of safety which is h greater or equal to alpha h. Okay. So just like some experimental videos where we deploy our >> Sorry Gary, I I have a question.

[00:35:23]>> Okay. Yeah.

[00:35:24]>> Uh yeah. Yeah, I was muted. Sorry. So yeah, in this video so so effectively what is happening is that the am I right that the the the human tele operator is just telling the robot to go straight but uh shield basically pick up okay I can't fully go straight because like I'm pretty confident it's going to bump into this valer and then therefore make adjustment while trying to adhere as much as possible to the original command.

[00:35:45]>> Yes. So the concept behind this kind of CC filters is that um as can be seen in this optimization problem here, right? I want to minimize my deviation to the original command while still adhering to the safety conditions, right? As you said, uh I'm just here pushing the robot to walk into me, right? But the robot with this lighter camera knows, okay, there's a human in front of me. I need to veer off the human, right?

[00:36:10]>> Okay, makes sense.

[00:36:12]>> Yeah. So these are just some like inab experiments where we deploy the system on both the built-in controller and our like just very hasty trained RL controller. This is just like literally the most basic RL controller you can find in today's like say MJ lab oraclab like demos right like as you can see with shield it completes this task with the nominal CBF. So without the correction and without the uh modifications we made to the safety constraints, it clips the item. And of course like if you just run a naive a star, it just goes through like through the item entirely, right? Like a star without the safety builder doesn't really care about like avoiding things like it cares a bit, right? It avoids a core of that thing, but the safety like radius, it doesn't really care about that. And similarly like this is just showing that it can handle like narrow gaps in this kind of systems where like this interesting thing here is that with shield since it knows when to be conservative and when not to be actually makes across this gap way faster than if I to run a nominal CBF we have to make it very very conservative for it to actually go through the item go through the gap. Yeah. And I guess my idea is always to deploy these humanoids. Right.

[00:37:23]So we just brought it outside with our perception system to just have it interact with people run around. It attracted quite a lot of crowd though that day and was like just writing the workshop that we had just show that oh it also works in this more closing environment. Same video and um we have some more other videos coming out. Yeah, this is actually my advisor's kid like um he's a very big Star Wars fan. So yeah, some others I can use the force.

[00:37:53]>> It's just the force. Yeah. Oh, don't go near me.

[00:37:56]>> Oh, that's >> Yeah, that's cool.

[00:38:00]>> It's fun to see the robot out like interacting with people like this. This is nice.

[00:38:05]>> Yeah.

[00:38:07]So going on to that paper, right? Like after we figure out, you know, how do we use this kind of uh things to make sure the robot aderes to our expectations, right? The next thing on that thread was how do you actually incorporate semantic information into this into this trajectory generation layer to generate uh safe trajectories that also have a priority, right? Like for example between like a human and a wall, the robot would rather hit the wall than a human, right? But in like nominal safety filtering like safety controllers, it fees everything as the same. and um it will just pick the random it will just pick a random position to escape if you don't have any like backup controllers behind it right if there's no solution just stop but if you don't do that just pick a random place to go so here the idea is if it comes to that at least you'll pick to pick to hit the wall rather than a human right like since the human is more valuable than the wall and a byproduct of this is actually we have also enabled uh social semantic guidance So for example, we always pass on the left of a human, right? Like when you overtake someone, it overtakes on their left and uh yeah and like whether you're facing that coming person or like taking over them from behind, right?

[00:39:24]Like and the way of achieve this is through a new method in our lab called poison safety functions. So essentially what this does is that instead of say we have different constraints for different humans, right? Like we have initially what we do is that for each table we have to we need to have one constraint right like um here what we do is we first convert this occupancy grid map into a unified poison representation of the environment and the good thing about this is that we have from the poisons all the gradients of that environment right so it's just one unified safety function yeah it's kind of like shifting the computation from the optimization solver that needs to handle like say 10 or 20 constraints to a paralyzable process which is like the poison solver that preconfutes this H and then the QP solver that actually handles the safety filters doesn't have to solve for that by any constraints. Um, Matthew is actually in this video. Um, she actually managed to push this onto the arm. So, the manipulator arm and sample points on that arm to essentially process like 100 constraints all together with this to enable a whole body avoidance on the arm. So, both of our papers will be at IROS this year in Pittsburgh. So, if anyone's interested, welcome to stop by. I think we'll all be there. Um, yeah. So again going back deployment safety needs semantic context right like geometric safety is semantically blind a person a chair and a wall occupy similar geometry but deployment should should not free time the same right so safety so our method makes a safety layer context dependent will keep the high rate control loop analytical so we do that given a trajectory we first say okay what's in the scene right so we first understand the semantics of the scene and then we generate the social semantic guidance field and And lastly the point on safety filter based on the guidance field. So the system is that we fuse point clouds and RGB images into semantic memory. Um since the RGB uh image here here we do have a gimbal but the gimbal is limited right. So we do associate semantic labels with a cluster of point cloud briefly for the short-term memory. So we build a semantic occupancy grid and then generate these fields and safety functions. So the the way we do the social semantic guidance field is that um since poisons you can give it initial conditions via a vector field. So essentially what we do is we manipulate the vector field to say okay if it's a human has needs to have more repelling magnitude and also if it's a human that's like going away from you or coming towards you that also rotates you also rotate the gradient around that the vector field around the human to be a counterclockwise or clockwise to essentially enforce this kind of sliding behavior right so it's only slide along this one direction yeah so as can be seen in the video here semantic margins change the safety field so we modulate the point on safety function by semantic class. It keeps wider margins around the human rather than static objects like walls or the boxes. Right? For example, on the left is a nominal safety filter.

[00:42:36]When I walk towards it, it essentially keeps the same distance between everything. But with our method, it gravitates towards the walls a lot more.

[00:42:43]Um, and also it also encodes the uh clockwise passing, right? Like for example, it I defaulted to me facing it and if it's I'm facing it our method even if starts on the left right knows to go to the right and sorry go to my left is right to walk across me right like but if this is normal one just chooses the closest thing which is just go on the left. Yeah. So again brought it out um actually across two campuses.

[00:43:08]So okay let me see if this video can play.

[00:43:13]There we go. Yeah. So this these three are like in our own campus where you know we show that can work hallways outside on a humanoid. This is the good thing about this is that it's very robot agnostic as long as it takes in C commands and on the upper right here we actually collaborated with one of our professors former students Ryan Coer who is actually a faculty at TUS now. So he put this onto their robot and ran this in the cafeteria for quite some time. I think half an hour or something just like having a walk around with like just people going around it. Um and I he told me I didn't really hit anyone or anything for that matter. Yeah. And people were like again as you can see right interested by the thing just going around in the cafeteria. No thank. So after we have passed through this a interface down into the trajectory layer. So given these planned commands right given these um commands into the controller right like how do we make sure the controller actually know safety. Okay so right so the last part is on policy learning which is how do we make sure that the learn controller itself actually understand what safety means. So this lead us to a paper from last year which is CBFRL that safety filtering reinforcement learning and training with control barrier functions.

[00:44:34]So RL again very powerful expressive can prior but they often prioritize performance at the expense of safety right like what we have in our lab is control barrier functions which offer a very principled method to enforce dynamic safety which is traditionally deployed online by safety filters however if we just slap on a CDF blindly onto any controller we see the fact that the policy having no knowledge of the CDF can lead to very conservative or very erratic behaviors. So what we do here is that we try to turn CBF convent interventions into learning signals. We integrate the control barrier functions into our training via both active filtering and rewards, right? Like both encourage more useful information gathering and also dense rewards on safety metrics because with the safety filtering the uh reward the learning tends to terminate less because for example in navigation environment bumps into obstacles less but then it also knows I'm in a approaching unsafe state.

[00:45:30]So my rewards got went down right and if I'm in this state what should I do because of the active filter. So at inference time this actually deployable without the safety filter running online because it has already instilled a notion of given this current conditions what should I do because I the reward was imitating the filter safety actions.

[00:45:49]So one issue with this is that if we actually want to run this in RL loop um with a vanilla safety filter we have to solve QBs at every step um and RL's environments steps also discrete right.

[00:46:02]So like how do we uh formalize this continuous domain thing into a more discreet hunt thing. So it turns out that we do not actually actually always solve QP. So for some problem we can set up very close home solutions and under certain conditions which is not surprisingly small enough delta t we can relate the continuous time safety conditions to the discrete time safety conditions and that's thus we can utilize the continuous time tools directly which means going back to the first point if we construct the constraint and the problem carefully we don't have to solve QPS we can just use a clos solutions yeah so system architecture is that we pass the policy through the CBF filter Um, and for the reward, we use both a more imitation-l like reward where we say we want the robot's own the policy's own produce action to be more aligned with the safe action. Uh, because safety filter, right? Like if it's already safe, it's just we if it's going to be in our safe state and vape interferes, then we actually want like the action to be like the safe action, right? And also we give it the safety penalty which means that if you are unsafe we penalize a reward. Um the thing with safety fil with safety function values is that in a lot of cases safe values don't mean much only the unsafe values do. So like something that's has value one which is safe versus something has value 10 which is safe. They are like essentially the same thing. So we don't want to encourage it to just stay extremely safe, right? alo do want to discourage it from being unsafe. Yeah. So an example we did is on navigation where we really quantify the results. Um here dual means that we apply the safety filter. Uh uh we we use like both the safety value reward and also the limitation for safe action reward and reward only is means that we only use a reward part. filter only means we only use a filter part and real time filter means that do we apply a actual filter online or not. So again as you can see if we do apply the filter both in training and online it essentially uh is the best performing one right like it reaches the goal 99% of the time and if we even add domain realization to the like dynamics because this is just a single integrator right it tracks the velocity so if we add like noise to that velocity tracking it also doesn't reduce performance and without a safety filter it's also the same like it's also is safer than the others and also doesn't really reduce performance if we add domainization. But all the others like reward only, filter only, filter only without realtime filter. As you can see, it's very bad because if the policies only train with a filter, but then you don't have the filter turned on like in deployment, it doesn't really know what to do, right? Because it's very reliant on the filter to give its um obstacle avoidance performance.

[00:49:01]And of course nominal is if we just don't do it we just like we just have a very nominal thing going on it's uh really also good. Yeah. So we apply this on human to task. So one is um uh there also back in the day right where climbing wasn't as developed as now I think right now the hot topic is really perceptive locomotion right we have like depth camera looking down at the stairs and informing the policy of what to do.

[00:49:29]All of this was done blind back in the day. Um so on the left here is we applied it to first the velocity tracking controller.

[00:49:38]So the user would prompted to like say walk into the boxes right but without actually applying a velocity filter on the user input the robot based on the positions of the boxes actually knows to modulate its joint actions to avoid the boxes. Right? Essentially run it own internal filter and on the stairs. So the we did some blind stair walking and I think a very big issue back then was the feet hitting the riser of the next stair. So we actually put a barrier on the riser of the next stair right. So like the feet that's the swing feet will never like hit it. And with this we're actually able to enable I guess better blind walking back then. But of course now with the uh depth camera actually looking down at the terrain you can do like I guess similar or even better performance without doing all those tricks. So one thing I was actually interested is that without the safety filter you see how it can't really cross this.3 meter gap but with it can um also it can go down. So yeah that was pretty cool and we we of course to like you know actually have it walk on real stairs. Um was really beat up there. So a lot of these like drifts is uh I think due to one of the motors being not really good. Yeah. So guess to close up this uh presentation my two cents is learning controllers or task specific policies is not the end all be all. Um and going back to the hand of paper like the quality of the data we feed these pipelines like these imitation pipelines is a real bottleneck for physical systems like we collect all the data we want right but how do we turn into actually usable data like actually good data is I think still needs a lot of good engineering and maybe good research and how do we go beyond the capabilities of language models and the vision language models to be more just enclosers for VA so for like world models right like um how do you actually use the say common sense encoded included to actually help us plan tasks and I guess the last question that I am thinking of a lot now is is there a perfect loop for robotics there is for language models and vision language models right is just going to be the data fly like the more data we collect the better we'll get or like we just can't physically collect that many that much data to be at the level of language models and vision language models because for those we have like accumulated data over like the past few hundred years for language models and like for video models it's like the entire internet which is of the same modality but for physical world data we don't really have the luxury now like do we just spend another 100 years collecting physical world data or do we find some uh structured way of doing it to lessen that data collection >> well do you think that things like these structured ways so you what you showed have been somewhat structured like a nice mix of data and structure. Do you think that there's a chance that that will bootstrap its way towards like these fully endto-end things because obviously that generates real data of robots interacting in the real world.

[00:52:44]That's what you just showed us, right?

[00:52:46]Yes, I think there could be possibility but the the really thing there is that I think generating physical world data um the cost of doing it is not the same cost as like say script internet or whatever like because with I think the closest analogy to this will be like video data right like the thing is I think with video data um it is you really collected so it's not intentionally collect Ed, right? Like people film themselves doing things before this I guess AI data collection uh craze was like they just film themselves doing things right like it can be it can be a family reporting that people YouTube it can be like seeing a movie right where like people just made for other things but uh coincidentally they can all be used for say video model training right like but for robot data the like the dilemma is I think the trick problem right like if it can't really do things um we don't want to we don't really want to put into homes to like act the masses might not accept them to actually like start the I guess everyone data collection trend right but then if we don't collect enough data we'll never be able to put them into homes and like put them into like everyone's houses to have them do this like data collection right like I guess one way that people are trying to look now is if you look at all those companies based off the wii structure like hiring like paying like say house cleaners or whatever to wear their heads setups and then when they going to like do their like normal chores they just wear the egocentric camera right like and then just record their egocentric videos and hope and hopefully turn them into robot useful data but then again it goes back to the point right the quality of the data there is a real problem like how do we turn that into actual like robot usable data is a very big open question I Yeah. So, >> yeah. Yeah. I'm I'm curious like uh Gary like you know your line of work, right?

[00:54:49]It seems like all tied together. I I was just wondering how far do you think we are from a demo where let's say you take a G1 uh you know let's say from your lab you give it a language instruction. Hey, I want you to walk across the entire campus which is one mile away. Walk across cross roads, cross streets with lots of people. Go up lift. Go down.

[00:55:11]Help me pick up this thing. come back to me. Um is is it is it are we quite near to something like that you think?

[00:55:18]Because you have the safety, you have safety, you know, guided uh policy. Um you have this very nice 10D controller and and of course like the agentic thing uh in in the first paper hand off, right? Uh do you think is is quite new?

[00:55:34]I guess if we talk, we have to like separate out the a demo versus like actually having this do it reliably and like like actually deployable like in a real world scenario, right? Like if it's a you know film demo, I'm pretty sure if I sink like you know a month, two months of like effort into it, just working on this every day, I can probably film a demo of like what you said like having it go across campus, pick up something and come back. But then like um extending it into actual like production right like actually having it be stable having it do this every time is takes a lot of like I guess engineering work um that's I think also a team effort right so just going back to the chisel model right like um there's a lot of like longtail problems in this pipeline that potentially maybe you can solve by yourself but there might be some things you don't think of right like there might be some things that can be better like modulized which if you work on it yourself you quickly get lost in the details and like you can't really look at the whole picture so I would say yeah demo wise maybe like half a year maybe a few months but like actually make this truly deployable like truly have it just roam around campus doing what people ask is going to take some time yeah >> but you but it sounds like what you're thinking is is actually all nearly all just engineering challenges not like fundamental research problems to do something like that >> I think This actually goes into another like debate that's I think currently going on is like in robotics right like is good engineering also science um I think it is because to interact with these kind of physical systems right like you always we always start I guess also I'm more of an engineering minded person like we always start with like defining a problem and how do we find good ways to solve this problem right not necessarily what cool new theory can I come up to like you know please the reviewers right like in this uh um I guess pipeline there will inevitably be things that may not be able to be solved with current engineering tools or may need some like twists and turns with current engineering tools and that's where the research comes from right like that's uh those like new problem need solving is where I think drives at least my personal research yeah but I'm in attempt at I think good engineering is also good research not just engineering Yeah, I mean very very true. I think 100% and and you know I I actually really like the the I think your the the the works that you line up in in this presentation you know I I feel that it's um it's makes it very I feel like it makes it very applicable you know in real world like you because like yeah you safety seems to be a very big part of your consideration and >> uh I rarely see controller that take into that kind of care into like what what safe means usually it's very performance oriented. Yeah, >> but usually the highly performant ones, you don't want to go near the remote robot, which you know actually means that you can't really deploy uh you know in the wild. So, so I think that's really cool.

[00:58:39]>> Guess yeah, preview into the another safety part is very cool. Um this is like what I'm working on humanoids like humanoid dog, right? Like just uh with a depth camera throw balls at it like play dodge ball with a humanoid. Yeah.

[00:58:55][laughter] >> Interesting. Wait, it's a uh but you the the neck is it's not too dark, right?

[00:59:01]It's still a fixed neck, right? Okay.

[00:59:02]>> Yeah, the neck is still fixed. Um we have a underground trying to design a like movable neck in the summer, but I don't think it's going to be in time for this paper. Um yeah, >> it looks cool.

[00:59:15]Okay, I was going to ask how many time have have the G1 been broken, you know, in this exercise, but [laughter] >> actually in this only once. That was a like personal record fixing I did last week where um I'm sure you guys know like with the G1 the biggest issue is the uh the waist signal cable like going from the upper body to lower body like the sim table is extremely thin. So if like if you rub it against anything it just breaks right. Yeah.

[00:59:43]>> Got it. Yeah. Um but yeah very very cool. Um yeah I think that's it. That's it for me Chris. Uh do you have any more questions?

[00:59:51]>> No that's great I think. Yeah. Yeah. No thanks I think. asked a few questions throughout. Cool work. It's really cool to see agents like humanoid robot agents and tools that would make them look like they would really be out there in the real world. The combination of the safety and agent work. Very awesome. So, I guess you want to move to our final question, Michael.

[01:00:10]>> Oh, yeah. The usual final question.

[01:00:11]Yeah. It's Did you come across any like cool papers recently you want to give a shout out to? Um >> Oh, yeah. Um I think it's the uh like vision language kinematics paper that came out yesterday and also the other one the other one from far I think the day before that they've been releasing a lot of papers these few days like on the like local the whole body local motion policy condition on like contact points that was also pretty cool yeah >> the one you're talking about is is it the kiny bl um >> let me see the first one give me a sec that's from a while ago.

[01:00:47]>> Yeah.

[01:00:47]>> All right. It's something else.

[01:00:48]>> It's it's this one. Let me tell the whole name. It's learning humanoid local manipulation from synthetic interactions in reconstructed scenes. So, VK >> actually, let me share the >> Yeah, this one.

[01:01:02]>> Oh, wow. Cool. Okay. I think I missed this one. Okay.

[01:01:05]>> Cool.

[01:01:06]>> And uh No, it's like it's extremely new.

[01:01:08]I think table came like either today or yesterday. Um >> the other one is this. Yeah, this one.

[01:01:15]So with this one like why I like it is also it's somewhat similar to what I did right like but they use contact points to prompt the motion tracker not like my 10D command so what they do is they like generate these like contact points online to prompt the robot right like for example feet has to be on the ground like hand has to be in contact like stuff like this yeah so it's another different form of like command interfacing but um I Yes, it's a bit more heavier than what I have in my paper.

[01:01:49]>> Mhm.

[01:01:49]>> Yeah. But >> yeah, I think what they are banking on is that if the tactile VAS do make a breakthrough and like VAS can generate the these contact points, then this is applicable to those kind of things, right? Like Yeah.

[01:02:02]>> Yeah. Very cool. Yeah. Okay. Maybe we invite them. Uh yeah, I think both are some from looks like it's at least from the same group.

[01:02:10]>> Yeah, it's from all from far.

[01:02:12]I think Karen anyway thank you so much uh for your time Gary uh you know thank you for for the presentation very well thought out uh you know I see the the line of thought you have you know the different project you take on it yeah um >> thank you >> yeah thank you >> looking forward to your next one yeah thank thanks for making the time >> thanks thank you >> okay all right >> bye