Stanford Robotics Seminar ENGR319 | Spring 2026 | Towards Trustworthy Autonomy

[00:00:09]It's really an honor to be able to speak here today. Uh you know I organized this seminar for almost four years. So long time ever since co and it's really cool to be able to take the stage now myself and deep dive into some of the work that I've done over the last few years towards trustworthy autonomy. And to set the stage, I want us to recognize that robotics is undergoing somewhat of a period of change where roughly 10 years ago, we were very accustomed to robots only performing these highly choreographed dances, repetitively performing the exact same motion over and over again in a very controlled environment.

[00:00:46]Today, robots are rolling out into the real world at scale. I want to emphasize that these systems are not prototypes anymore. We are actively scaling these systems. For example, Whimo is growing its operations so quickly, they are now doing almost 500,000s of rides per week uh in the United States.

[00:01:06]And this progress, I would argue, has largely been driven by advances in learning algorithms. We're increasingly integrating learn components into all aspects of the stack has allowed us to make sense of complex sensor modalities and allowed robots to make the right decisions in the messiness of the real world. like this crazy example of a moped that crashes right in front of the car. With the emergence of fully endto-end AI architectures, we are now also seeing particularly promising results in general purpose manipulation systems, raising the prospects of robots helping in our workplace and our homes.

[00:01:41]And this all means that in the coming years, robots may have a profound impact on the way that we move, the way that we work, and even how we live our daily lives. But such scale also comes with safety risks as our systems are still far from perfect. Those amazing self-driving systems that dodge accidents also tend to drive straight into a flood and sink when it rained in San Francisco. They may mistake the sunset for a traffic light or even stop in the middle of the freeway because the car drove past a billboard with a picture of a stop sign on it. And similarly for manipulation systems, for no apparent reason, they may mess up your cooking.

[00:02:25]They may stick plastic utensils into the oven instead of the drawer that you asked it to put it in or wreak havoc on themselves and their environment.

[00:02:34]And when we go towards these blackbox systems for learning all of these behaviors, it becomes really hard to tell where are these failure modes coming from and what can we do about them. And that has been the overarching question that I've studied over the last few years.

[00:02:51]To overview some of the work that I've done, let's first build an intuition on where these failure modes are really coming from. Right? The systems we have today are not magic. They are datadriven systems. And to develop them, we collect a lot of data, label it, and train a model to match these training behaviors.

[00:03:08]And this process has some inherent shortcomings, right? namely that a model is only going to be as good as the data that it was trained on. And so eventually when you deploy a robot in the wild, it's going to encounter these things we call out of distribution inputs, things that were very dissimilar from that training data on which models may perform very poorly. Examples of this can include longtail cases, situations that are so rare that they were not well represented in training data or scenarios where the distribution of input suddenly shifts like when we train a model on daytime data only and then suddenly the model gets deployed at night.

[00:03:44]Fundamentally, any model is always going to be imperfect because we train it to pick up correlations in the training data, not true causation. And so they're prone to pick up correlations that are spurious that don't hold in general.

[00:03:56]like this famous example where a classifier has learned to correlate an animal's type with its typical background instead of learning what really makes a cow a cow and therefore in my view require guard rails to cope with the practical reality of deploying learning enabled systems in the real world right eventually you're going to encounter these out of distribution examples in the wild and besides tools that improve and validate our systems on known unknowns we also require runtime monitors to get guard against the out of distribution unknown unknowns the things that are rare and we couldn't anticipate at design time and rather than hoping to build models that are always perfect my work is instead focused on building systems that I would say are trustworthy systems that know what they can and cannot do that recover safely when conditions become too challenging or hazardous and whose model behavior we can imp interpret diagnose and improve when failure modes emerge.

[00:04:56]To do so, my work has focused on building runtime monitors that detect when models are unreliable, safety interventions that avoid failures, and a data flywheel that uses devel deployment data to improve system robustness and performance over time. All my work falls in these three categories, but today I will go into through uh into detail in two papers with a two-part focus. The first is on building systems around the models that we deploy to improve safety at a system level. And the second is on actually improving the models themselves through a deployment data flywheel. In the first part, we're going to focus on a particularly challenging notion of safety that I'm going to introduce and call semantic safety. So digging into the first part, the concept of semantic safety all started when we came across a set of realworld robot edge cases.

[00:05:49]Particularly when we saw this self-driving car that got confused because it was driving behind a truck that was carrying inactive traffic lights. We saw this car stopping for a person wearing a t-shirt with a stop sign on it. or this example where a robot is placing plastic utensils in the oven instead of the drawer that it was asked for. And these failures are kind of complicated because they arise from the holistic context and the interreations of objects in the scene.

[00:06:17]It's not simply about avoiding obstacles anymore, right? And so we call these failure modes semantic anomalies, unusual situations where ordinary objects and the overall context together lead to some system level confusion in the stack. And when we saw these cases, we wondered what would existing outofdistribution detection methods do which have really mostly studied the outofdistribution problem through the lens of physical safety like obstacle avoidance. And existing methods that, for example, measure visual novelty with respect to the robot's training data or bootstrap uncertainty scores on a model predictions didn't really work at catching these kinds of semantic anomalies because there's nothing that's really inherently visually novel about seeing stop signs, even if it's on a billboard. And the model is extremely confident that what it's seeing is actually a stop sign. Right? These things are system level errors that defy component level blame assignment. and the response depends somehow on the context of the scene. So how can we safeguard against semantic anomalies?

[00:07:22]The core idea that we are going to explore is whether we can leverage the common sense reasoning capabilities of large language models to reason about the context holistically with the rationale being that the internet scale pre-training data that these models are trained on brings many lifetimes of experience to the driving task that endows models with a common sense to mitigate semantic anomalies.

[00:07:44]So now I think you might say this is quite reasonable. I can believe that a model like GPT5 can can handle these kinds of things. But how do we operationalize it? Right? How do we practically increase closed loop robot trustworthiness with language models?

[00:07:59]Doing so requires solving two key challenges. The first is to mitigate the computational cost of language models to enable a level of reactivity. Right? The best language models are very large and that makes them extremely slow. is not very useful for a robot that moves very quickly. And the second is that besides detecting anomalies, we now need to integrate these reasoners into the control of dynamic and agile robots.

[00:08:23]We propose a two-stage decision-making framework to address these challenges.

[00:08:27]And to get some intuition on this, let's first think about why LLM are so slow.

[00:08:31]And it's really because they generate text by sampling tokens one by one auto reggressively, right? And this auto reggressive process is really the bottleneck. We need to call the LLM's backbone maybe hundreds or thousands of times to generate a reasoning trace that allows a model to figure out what to do in unusual context.

[00:08:52]And so we propose a two-stage reasoning pipeline where the first stage leverages a intermediate outputs like single embedding coming from a model backbone to enable reactivity. But we still rely on the full generative chain of thought capabilities of the largest models to make zeroot decisions on out of distribution scenarios that have never been seen before.

[00:09:13]To do this, there is this notion of an embedding that we can back out of a model's forward pass. That is when we take a input text and try to produce an output text, there is an intermediate vector that we can take out of the language model. And when we graph these vectors in a sort of stylized latent space, what we have observed many times in literature before is that these embedding vectors preserve the semantics of what has been encoded by the model.

[00:09:40]So concepts that are similar like uh you know two types of animals are going to be close by whereas concepts that are very different like a truck is going to be far away in this embedding space. And so we're going to detect semantic anomalies via similarity queries with these embeddings. To do so, we first assume that we have a data set of prior experiences of the robot like the training data that the model was trained on. And we use a foundation model to generate a database of these semantic embedding vectors.

[00:10:10]Then at runtime, we take the current observation of the robot, push it through the embedding model again, and compare its similarity of the current observation with the embeddings in the database. If the current observation is similar to the previous experiences of the robot, we call that nominal. The robot has probably seen it before and we continue with the decisions that were made by the base autonomy stack. If the observation is very different from what we've seen before or anomalous, we're going to query the full generative can of chain of thought of a large model to reason about what safety preserving intervention to take.

[00:10:48]So how do we integrate this fast and slow reasoner within a planner? Well, the first is that we construct recovery sets which we're going to call control invariant subsets of the state space that correspond to a high level safety intervention and we provide these sets as fallback choices to the LLM in a multiple choice fashion. So for example for a drone delivery service you might say that a recovery set could be landing in a field, landing on a rooftop or hovering in some kind of holding zone, right?

[00:11:17]And we then use these recovery sets within a model predictive controller that maintains a tree of fallbacks associated with the nominal goal and a set of safety interventions. We can compute embeddings of the observations in parallel to detect anomalies and we show that this can be done in real time.

[00:11:35]When an anomaly is tri detected, we provide the current fallback options to the LLM reasoner. And as shown in orange here, you can see that the fallback trajectories overlap for an upper bound on the time it takes the LLM to return its decision. This ensures that the fallback options that we provide to the LLM are still dynamically feasible once the model returns its decision, allowing the MPC to account for the time it takes the model to to reason and safely enact whatever the LLM decides.

[00:12:08]We can prove that this controller design allows us to reach the recovery set chosen by the LLM with a hard guarantee.

[00:12:16]And as a case study, let's reexamine this stop sign on a billboard example. A base autonomy stack detects the stop sign on the billboard as a real stop sign and slams on the high on the brakes on a highway on-ramp, which I don't think is very safe. If we look instead at our approach, the vehicle is first driving normally but fires off a query to a language model incurring in encoding the current observation once an anomaly gets detected.

[00:12:45]Now this embedding based similarity query only requires one pass through the network, right? And so that's something that we can do very fast like 20 hertz on an Nvidia Jetson. The planner has this emergent behavior that it then slows the car down while waiting for the LLM to output budgeting time for the reasoning process. And in this case, the LLM decides to keep driving, ignoring the billboard without ever stopping the car.

[00:13:11]And while we can detect anomalies quickly with small models, it's necessary to reason about the downstream consequence of anomalies with larger models. As the sort of table here shows, larger the model gets, the better the reasoning becomes.

[00:13:26]Quantitatively, we test these anomaly detectors on a series of synthetic data sets where we programmatically create descriptions by drawing from sets of nominal and anomalous observations in multiple robotic domains. Right? For an AV stop signs are nominal, elephants are strange. Manipulators may seek forklifts near them driving around in a warehouse often, but knives on the on the conveyor belt is probably uh a problem.

[00:13:54]And we evaluate a number of embedding models of various sizes on these anomaly detection tasks where we can see that all these anomaly detectors reach very high accuracy as the size of the embedding cache increases. And some takeaways to point out here are firstly that grounding these anomaly detectors in prior experiences of the robots with embeddings actually is able to outperform generative reasoning at the anomaly detection task. You can kind of see here that the the horizontal lines correspond to just querying GPT models to reason about whether it's an anomaly and the embedding detectors are actually doing better. Right? The second is that smaller models can do just as well as larger models at the anomaly detection task. We can see that small models like say MPNet and BERT which are like roughly 100 million parameters are actually getting extremely high accuracy at detecting all of these anomalies. And this is not magic, right? To really understand what the embedding based detectors are doing, we're also going to do an sort of nonID sweep where we hold out concepts from the nominal database and slowly start adding them back in.

[00:15:03]Right? And what you can see is that the accuracy increases roughly linearly with the percentage of nominal concepts in this database. And the takeaway of that is really that this embedding detector is detecting differences with respect to prior experiences. If we remove highle concepts from the database, they start being detected as anomalies. Right? And this means that the fast and slow reasoners are both necessary in this fa framework. The fast reasoner is what gives you reactivity. We can use mpnet and run it at 100 at 40 hertz on an NVIDIA Jetson. Whereas the slow reasoner or the large models are really necessary to produce the long reasoning chains to accurately assess the safety criticality of an anomaly.

[00:15:49]We also deploy these reasoners on actual quadrotor hardware where for this task the nominal goal of the quadrotor is to land on the red box in the presence of a clutter of inconsequential objects. And we see that here on the right, if we place other quadrotors on the boxes, the drone correctly detects this and goes off to the holding zone instead. And what's interesting about this case to me is that quadrotors in general have been seen plenty of times by the robot before. What is really making them detected as anomalies is that their context has changed when we put them on the landing zones and the boxes.

[00:16:27]Similarly, on the top example here, we're going to place a previously unseen object on the ground, in this case, a keyboard. And we see that the drone correctly flags this and starts slowing down while waiting for the LLM to output. Whereas on the bottom video, the drone immediately backs out of landing when it detects the other quadrotor on the red box. In the case of the keyboard, the top video, the drone confidently proceeds to land on the box, whereas the other drone on the red box forces the LLM to output landing on the blue box instead and diverts its decision based on the context of the scene.

[00:17:03]A quick recap of these methods is that we can build safer systems with runtime monitors and safety interventions where we introduced the problem of semantic safety contrasted with more classical notions of physical safety. Highlighted the effectiveness of foundation models to detect semantic anomalies and increased the robot's closed loop trustworthiness via thinking fast and slow system design.

[00:17:27]Following these works, our approach has found broad utility across robot platforms like hazard identification in construction robotics, runtime monitoring of marine autonomy, similarly to space autonomy applications and even a wide variety of use cases detecting fail and preventing failures in manipulation.

[00:17:48]With that, I want to conclude the first part of this talk and move towards the second part on guardrails for end-to-end policies. And what I want to focus on here is that while runtime monitors can safeguard us against failures, I think big question that we have is once we have seen these failures, how can we systematically improve our models?

[00:18:08]And let's first dig a bit deeper into what robotics models are really looking like these days to make an understanding of how to proceed. We see this trend where we're increasingly moving away from modular architectures where each model completes an interpretable subtask like perceiving its environment you know making a prediction of what's going to happen and then deriving an action towards monolytic AI models that take in observations and directly produce actions taking advantage of progress in robotics adjacent domains like NLP and training large generative models on increasingly large data sets a scalable recipe to teach robot many skills.

[00:18:49]But while prior work shows that we can detect failures, we don't have any handle on why these big monstrosities fail. There's no control logic that we can adjust when our end-to-end policy lacks the precision to tie a cleat, chooses a fragile pushing strategy when tucking a box with uncertain mass under a shelf, or when it sometimes completely ignores other objects like dropping books from the shelf. The key challenge is how to debug these blackbox models to systematically improve performance.

[00:19:22]And again in this context I'm going to advocate for a datacentric view. Our policy is only as good as the data that it was trained on. You know it can be the failures can occur because of the mixed skill levels of the human demonstrators. Teleoperating a robot is not easy, or by some diverse strategies in the demonstration data being collected by many people, some of which are just inherently more fragile, or because long-term data collection efforts might contain spurious correlations, like a janitor that may have changed the background of the robot here in a way that weirdly correlates with the behavior of the robot. These underlying root causes are so varied that it becomes very difficult to systematically identify them within large data sets manually. We need automated tools to trace the behaviors of the robot back to the quality of the and the composition of the robot's training data.

[00:20:16]Having thought a little bit about end-to-end systems and their failure modes, let's think about datacentric policy improvement.

[00:20:25]Concretely, suppose that we train a policy via behavior cloning and then evaluate its closed loop performance.

[00:20:32]Qualitatively, what we want to understand is what training demos drove this test time behavior and created these failures. And how do we even formalize this goal between collection of demonstration data, training a policy via maximum likelihood, and then actually testing the policy in closed loop. Our key insight is to develop methods that counterfactually reason about the impact that inclusion of or exclusion of the training data has on policy performance. That is our goal is to predict things like how does the expected return or the success rate of the policy change if we remove some demo from the training data. A process that we're going to refer to as data attribution. And instead of training policies on all possible subsets of data that we have, we want to do this in a predictively a predictive fashion, right? We want to predict these counterfactuals. To do so, we turn to the theory of influence functions, a foundational tool from the robust statistics community that traces its origins back to at least the 1970s. And these tools have really formed a foundation of interpretability research in deep learning since the seminal work by Coen Leang in this building.

[00:21:44]What are they doing? Well, influence functions effectively compute a first order sensitivity around reweing a training sample. Specifically, consider the following training objective. We train a behavior cloning policy by minimizing the average loss over all the demos that we have collected. And consider a perturbed objective where we take one of the losses on one of the samples and add an infinite decimal importance weight epsilon on the loss of some demonstration towel.

[00:22:17]What we want to do is compute the influence that that importance weight has on the policy's performance. that is what is the change in expected return of the policy with respect to up or down weighting this particular sample of interest through the importance weight epsil epsilon. So this is a sensitivity that allows us to interpret how the performance of the policy depends on a sample of interest right upweing it or downweing it might increase or decrease the performance of the policy. So all the equations aside, it's a measure of how the training data relates to the test time performance.

[00:23:00]So how can we actually compute this magic quantity? Because I wrote a bunch of complicated equations on the board.

[00:23:06]Yeah, it turns out that you can directly compute the influence function in a typical machine learning setting where you make an individual prediction and have an explicit equation for the loss of that sample. But in our setting, estimating the performance influence requires reasoning over the outcomes of sequential decisions. Whereas existing tools built by the machine learning community only attribute individual predictions that a model might be making. And we cannot directly compute these quantities to begin with because it depends in robotics on an unknown reward function and the environment dynamics. Right? I roll out the policy and I see whether it succeeded or failed. I have no way to mathematically write out uh all the steps of of what it really means to complete some of these tasks. And so our insight has been to apply policy gradient tricks to decompose the performance in influence into a tractable sum of the influence of training data on the action log likelihood of the policy.

[00:24:04]These things we can easily compute with existing tools like track which is a tool that uses matrix sketching to compute reduce compute costs. So essentially we're going to average over rollouts of the policy quantities that we can compute basically the sum of all of the influence functions on individual actions. And all the mathematical complexity aside, the takeaway is that we now have an estimator that can causally relate training data to deployment time performance.

[00:24:33]This is something that we can use as follows. We can train a policy, test it by doing some rollouts and then estimate how the training data contributed to that performance by computing the performance influence. We can then use this performance influence to curate data that most positively influences the policy's deployment time success. Right?

[00:24:55]We can remove lowquality data using this tool or choose the most valuable newly collected samples to add to the data set resulting in a data flywheel where we train a policy, evaluate it and then systematically curate data that improves the policy's performance. We're going to call this algorithm Cupid.

[00:25:12]And in experiments, we show that this allows us to prune lowquality samples, identify the most robust train strategies in training data under test time distribution shifts, and even allowed us to root out spirious correlations in a case where we collected a bunch of data and actually pruned twothirds of the original samples away. And by pruding we can go from 40% success rate to 90% success rate with 2/3 less samples fewer samples because the samples had some kind of problem with it across all of these tasks.

[00:25:51]In [snorts] contrast, if we look at some of the baselines like a method called demonth, which is more of a huristic that measures how noisy the trajectories of the robot are, which is sort of a intuitive measure that we as humans cooked up to think about data quality.

[00:26:05]We find that these heruristics may work really well on some tasks where actually noisiness in the data is really the cause of data quality problems.

[00:26:14]You know, it may work well in in some settings, but actually sometimes it may be anti-correlated with how a model really learns, right? Because our intuition of how models learn may not really match with the reality of what the model is picking up from its training data.

[00:26:30]We also have applied this beyond a simple single task diffusion policy setting to curate post- training data for VAS where we select a small fraction of the data from an existing pool using cupid and actually see that it vastly increases the post training performance of a VA like PI0.

[00:26:51]And so to quickly recap, uh, in this part of the talk, we looked at how to improve models with a deployment data flywheel. We proposed a method to causally relate training data to downstream KPIs and developed an algorithm to systematically improve policies via data curation.

[00:27:10]With that concluding the second part of the talk, I'll quickly wrap up and talk about some future work.

[00:27:17]Right? The key challenge that I've looked at in in in the methods that I presented here today is that the models that we use in robotics are only as good as their limited training data making the risk of out of distribution edge cases ever present. The guiding principle that we should take in my opinion is a safety first process that links development and deployment tightly together. Building systems that users can trust rather than models that are always perfect. And the approach for doing so is to create guardrails that detect impending failures, avoid their negative consequences, and diagnose deployment behaviors to transparently improve a model's performance.

[00:27:57]If you're going to go home today, there's really two things that I would want you to walk away with, right? which is first this view that runtime monitors offer a pragmatic paradigm to detect failures, avoid their negative consequences and improve the long-term performance of learning enabled systems rooted in unanticipated and anomalous conditions. And the second is that taking a dataentric view allows us to understand learning systems. What is in the data? What is not? And how does that data drive performance? These are the questions that are very helpful to understand what's really going on in deep learning systems. I'm not going to have too much time to talk about future work, but uh I would like to point at a perspective piece that we wrote at the start of my PhD that sort of outlined many of these challenges, including many opportunities for future work.

[00:28:48]Um I'm just going to leave this up at the slide and thank my collaborators and I'm happy to t take any questions.

[00:28:59][applause] Yeah.

[00:29:05]>> Yeah. Um really interesting talk. Um what do you think the bar is for the um semantic unsafety description?

[00:29:18][clears throat] I saw the accuracy four.

[00:29:28]Yeah, this is a great point. So your ju just for uh I don't know people on on Zoom, the question is you saw that these uh language models are not perfectly accurate at detecting the semantic or reasoning through the semantic anomalies uh quite yet and so how many nines of reliability are we away? It's definitely true that these models are not perfect yet. But I think what's clear is that as time is passing and these models are getting better and better and better, the accuracy is improving quite quickly.

[00:29:59]I think it's the most promising path to to go because of uh sort of two reasons. The the kinds of errors that these models are now catching were just situations that no other method that we had before was able to detect even to begin with, right? And so even though it's not perfect yet, you can see the field broadly investing in reasoning models for autonomy because it's unlocking new capabilities that weren't there before that are there to increase safety. The second part is that in that particular work, we are using the foundation models in a sort of additive fashion that are not interfering with the base behavior of the of the model. Right? And so if the if the LLM is reasoning that uh something is unsafe, it can only ever make the system more safe, right? Because uh you didn't know how to handle it to begin with. Um and if the model, you know, has a false positive, it only makes the system more conservative. It's nuisance for a user, but it's not going to create additional hazards, right? here is sort of using it to add more to the things that you you can handle if that makes sense.

[00:31:20]Any other questions?

[00:31:24]>> Yeah.

[00:31:25]>> Oh, so it seems like a lot of your metrics were for like zero shot or like one attempt at completing the task, but by curating like bad data, does the performance decrease if like the policy tries to attempt a task multiple times?

[00:31:41]Let's say it's trying to wrap the rope around the anchor. Like if it fails the one time, will it decrease its chance of succeeding in time?

[00:31:51]>> So you're saying like if I curate out that data, maybe it like you know reduces the ability of the model to do retries. Is that what you're saying?

[00:32:02]>> Um that that's a good question. Um the met the way that these methods work is that you define the metric right like that's the job of the designers to identify the right metric that you want to track and you want to hill climb on and in this case what we used was does the policy tie the cleat within uh I would say one minute period right and so if there is data that is not contributing to the model completing the task in a one minute period then that data is probably flagged as bad data and and removed, right? And so I would rather encourage my model to complete the task quickly than to encourage it to retry endlessly, right? So with the time limit that we set sort of says that at some point retrying over and over again, we're going to count that that as a failure, right? Um it it's hard to say whether re you know keeping retry behavior in general is like a type of data that we really really want to add in in general because we have no like human our human intuition about what what's good data is like really not useful in like predicting whether a model is actually going to get better.

[00:33:18]And so these tools that actually causally back out whether the performance is increasing or decreasing are like a bit more general.

[00:33:29]Yeah.

[00:33:30]>> Yeah. For the first part of your presentation, I was curious to know if you had tried out any other models to the fast anomaly detector. For example, something like a just the image observation and the text from your database or something like that. Yeah, we tried a number of um models as I sort of showed a a hu you know huge list, right? And uh generally what we found is that uh the performance at anomaly detection sort of high across almost any model that we tried. And so for the purposes of increasing reactivity, we just press the latency as much as we can and go with the smallest thing that we could possibly um possibly find, right? And and that sort of hints that the anomaly detection task is a somewhat easier task that small models can do even though it's not sufficient to actually be safe, right? Because it's only telling you if something that you're seeing is different than what you saw before, not necessarily if it's dangerous.