Jorge Mendez-Mendez: Unlocking Lifelong Robot Learning With Modularity (2023-10-05)

[00:00:00]super happy yeah today we're super happy to have uh Dr J Mendes as our speaker um Dr Jo MZ is a post-doctoral fellow at MIT ciso um he received his PhD from the grasp Lab at the upan uh his research focuses on creating versatile intelligent embodied agents that accumulate knowledge over their lifetimes leveraging techniques from transfer and multitask learning modularity compos compositionality reinforcement learning and task and motion planning so his work has been recognized with an MIT IBM distinguished post-doctoral Fellowship a third place of the two Sigma PhD diversity fellowship and the best paper award award in the lifelong machine learning workshop at icml so without further Ado let's welcome Dr Mendes so when every household has a home assistant robot it is clear that these robots will have to be versatile and adaptive we will expect these robots to have many capabilities and each of us will expect our own robot to adapt to the particulars of our home the chorice we need from them how we like them done the tools and appliances at its disposal and once's more our preferences and needs and environment are likely to change over time it is clear that the robot company won't be able to anticipate all scenarios each and every one of our robots will ever face in in our households so the robots will need the ability to adapt over time by learning from data so for example you can imagine the robot is going to use reinforcement learning to learn to type tid up our office so it might use some amount of data to learn to move the chair out of the way a bit more data to learn to get close to the table Gras the folders and so after using a whole bunch of data to learn each of the parts of the task we get our office sorted out and this seems pretty good if the agent later wanted to use reinforcement learning again to learn to sort these books the naive solution would be for it to start all over again so it would use some data to relearn to move the ch there and get close to the table some more data to learn some new things like grasping these heavy books and so after a whole bunch of new data the robot learns to sort our books and this no longer seems that good if we wanted to have a general home assistant robot it would require lots of data to learn each task one after the next and the problem is that the robot is not reusing any data from those earlier household chores to reduce the amount of new things it needs to learn to solve later tasks and this is the Vanilla single task machine learning setting Knowledge from one task is used for that task and that task only and never transferred in any way to any subsequent task If instead the robot was able to leverage its knowledge about navigating to objects and picking and placing stuff we could go from this picture to this where the robot progressively becomes more efficient at handling new tasks by building upon its accumulated knowledge so what we need are robots that are lifelong Learners that accumulate knowledge over time to become progressively more efficient at handling new problems these types of methods would enable robots to become substantially more versatile the key Insight that I will share today and if you will take away one single message from my talk it should be this one slide is that modularity is the key that is going to enable these capabilities because robot problems are modular and modularity makes lifelong learning fundamentally easier let's start by looking at some of the advantages that modularity brings to lifelong learning the robot is going to face the problem of treat learning tasks in sequence one after the next it's going to use any data it collect from each task to accumulate knowledge and the metric that we will use to measure how well it accumulates knowledge is the efficiency of the robot at learning to solve each new task one key requirement that algorithms will need to meet in order to to achieve this accumulation of knowledge is to avoid forgetting the knowledge that they have uh updated in the past so for example int intuitively the robot might choose to very rapidly adapt to the new task and overfit to it but that would lead it to forget the knowledge that it had accumulated in the past and that will prevent it from reusing that knowledge in the future so for example if this robot accumulated some knowledge about opening doors and moving around groceries and it decides that it will scratch that in order to specialize very quickly to opening these boxes the robot will have to relearn all of that information about doors and groceries if it later has to face another task that requires that the typical solution used in most of the literature today is to use one giant neural net to contain all knowledge about all the tasks so for example if we train this robot in the kitchen task it might contain some knowledge about opening doors picking boxes milk cartons bottles all mixed up in this giant network of Weights if the robot then wanted to learn about opening these boxes and it knows that it should avoid forgetting so it wants to make sure it retains this knowledge about doors in case it's useful in the future the problem is that it doesn't have any way to know where in the massive tangle of Weights the knowledge about doors is stored and so it is doomed to update all the weights and potentially override some useful knowledge modular systems give us three key advantages for solving this problem the first one is that knowledge about different parts of the task is encoded in completely isolated modules so information from one task has no mental effect on modules that are not relevant to that task the second is that the agent can progressively increase its repository of modules over time to handle a broader set of tasks and the third is that the robot can combine its existing modules in new ways to solve more tasks compositionally let's now have a look at some of the ways modularity arises in robot problems and we're going to focus on two forms of modularity at the temporal level the agent can decompose its solutions to problems into high level abstract action say navigate to an object pick it up place it and at a functional level the robot can decompose the solutions to the same problems into stages of a processing pipeline perceive the scene plan for how to solve the task control the robot itself so let's have a look at what these look uh at these two with an example we might have this orange arm here in charge of picking up the box and placing it in the Target location to its right so it might use some temporal modules like picking up the block lifting it off the table and then placing it down on the right if we then wanted to solve a similar task with the robot arm like placing on the Shelf it can Leverage The picking and the lifting module use a separate module for placing on the side and immediately solve the task but if instead we wanted to solve the same task with this black arm here there is nothing about these temporal modules that would allow the robot to solve the task because they have become specialized to the orange arm If instead we had a module a functional module for controlling the orange arm and we replace it by a functional module for controlling the black arm we could leverage the detection of the blog the planning for the task and immediately solve the task so we have seen that robot systems and the solutions to robots are robot problems are modular that modularity gives us a lot of advantages for training lifelong agents and that lifelong learning is the key toward developing versatile robots throughout the rest of this talk we're going to go over two solutions that build off of these ideas the first one uses neural Nets to represent each of the modules and the solution to a task is a modular net uh that stacks these modules one after the next and the agent is going to be in charge of learning the interfaces between modules so the output of each of these modules in some abstract Vector space the second solution is going to use neural Nets to learn the elements of a task and motion planning system where the T system itself is going to dictate how one module communicates with the other compared to this T based solution the neural Net Solution makes fewer assumptions about what each module should compute exactly but requires lots of training data to figure that out to learn those interfaces between modules the temp system leverages additional engineering effort to reduce the amount of data required to learn we're going to start off by looking at the neural net case where we're going to establish some of the advantages of modularity and compositionality and then we're going to come back to the neuros symbolic case with Tamp so in this first part of the talk our goal will be to develop re modular neuron Nets that can solve multiple robot tasks if we succeed we should see that the agent learns faster because each module is trained on data from multiple tasks and that it can solve new tasks that it didn't train on by combining its modules in new ways the task that this system should be able to solve are things like this we have the orange arm in charge of placing the block on the shelf and then throwing the same Block in the trash then we can have the same two tasks but solved with a different object like a plate and then we're going to have a different arm that is in charge of solving all the same four tasks and the way we're going to do this is we're going to use one neural net to learn the grass post for the block and one for the plate another neural net to learn to plan the trajectory for throwing in the trash and one for the shelf and then another neural net for learning to control the motors of the orange arm and one for the black arm and then we're going to stack all these together into one longer net where the output of the grass pose detection module is going to give us a waypoint for the trajectory planner the trajectory planner is going to give us a path in rer Bros space for the joint space controller and the joint space controller is going to drive the motors of the robot so more generally we are going to assume that problems can be represented in terms of a compositional problem graph the idea is that the robot can solve a problem by applying a chain of functional Transformations all the way from the input observation to the output action so this doesn't really say much this is true for any neural net policy where each of the layers is one functional module so the second second assumption is that the with agent will be able to combine the modules in multiple ways to solve more tasks so if we can solve task a with these two modules and task B with these other two modules this assumption says that we can solve task C and D by combining these modules in new ways the compositional problem graph is a directed graph with two types of nodes the first type are random VAR representing the inputs and outputs of the task so we have an input note with in degree zero that represents the state space of a task and an output note without degree zero that represents the output space of the task and the second type of note is going to be the functional Transformations such that for an edge from F1 to F2 F2 is going to take as input the output of F1 the solution to a task is going to be a path through the graph from input to output and the key notion here is that any function that lies on two paths in the graph is going to be part of the solution to the two problems corresponding to those paths and then once we have this graph in place we can use neural Nets to learn each of the functional nodes in the graph and the solution is going to be all the neural Nets in a path stack together one after the next which is equivalent to the functional composition so in our example we have modules for detecting the grass post of the block and planning the trajectory for the trash can and so on and because of this sharing of modules across tasks we can use these six modules to solve eight tasks and this is just a toy example the number of tasks we can solve grows multiplicatively in the width of this graph and exponentially in the depth so we have these neural Nets the question now is how do we train them we're interested in a system that faces tasks sequentially one after the next but we're actually going to start by looking at a simpler problem where the agent trains on a bunch of tasks all at the same time because if we can't solve this easier version of the problem we stand no chance of solving the harder sequential version and in this setting we're going to give the agent full access to the actual graph so it knows what task it is currently solving and it knows what path in the graph it needs to use to solve the task but it's going to be in charge of learning the functional modules so the interfaces that communicate between one module and the next and we're going to train the whole system end to end via reinforcement learning so we're not going to give any intermediate supervision about what each module should compute so the first question is do we really care about the agent learning the right thing the functions that we set out for them for the modules in the composition graph and the answer is that we definitely care because that is required for compositional generalization imagine if we trained the orange arm to place the block on the shelf and it somehow ends up with a different functional decomposition maybe the grass post module learned something specific about the orange arm and the Shelf trajectory learned something specific about grasping the block that would prevent us for from solving the task with the plate because now the trajectory for the Shelf is entangled with the block grasping and so the second question is how do we expect to learn the right thing if we're just maximizing rewards and the key to that is to use lots of data or more precisely to train the system on lots of different combinations of the modules so we're going to train the agent to throw the the Block in the trash with both the orange and the black arm and to plan the trajectory for grasping the block with both U for both throwing in the trash and placing in the shelf and that is going to prevent these systems from specializing too much so for example the the the placing the block on the Shelf can't specialize to the orange arm because it wouldn't be able to maximize rewards for the black arm if it did that so we're forcing the system to learn the right thing by having it train on many different combinations of modules one major Ro block for evaluating these ideas is that there is no good reinforcement learning playground for robots that has these very explicit compositional relations between tasks so I want to talk to you briefly about this Benchmark that I developed with my collaborators the key idea is to leverage our compositional problem graph to construct a large set of robot tasks we saw that the number of tasks that we can build Bros with the width and depth of this graph where we can think of the depth as being the number of aspects that change from one task to the next so the robot arms the objects the task objectives and the width as being the number of variations of each of these aspects so how many robot arms how many objects so like we've been doing we're going to vary the object the robot arm the task objective we're going to add obstacles as one additional aspect that varies between tasks and we're going to have four different variations of each of these aspects so we come up with four to the power for reinforcement learning tests and by the way this is all built on top of Robo Suite out of I think Folks at Stanford so what we get look something like this we have this orange arm that learns to pick up the block and throw it in the trash and then we can have the black arm do the same task throwing the Block in the trash then we can change the task itself and have both robots place the block on the Shelf instead of throwing it in the trash we can change the object and have both systems do the four tasks with the plate that requires grasping from the side instead of from the top and then we can add obstacles to the environment that limit the free space for the robot what's really powerful about this idea is that we can easily vary a handful of things about the environment and we come up with hundreds of reinforcement learning tasks Each of which requires a meaningfully different Behavior but but in a way that we understand exactly what changes from one task to the next there are two open source versions of this Benchmark one for online training by interacting directly with the simulator and one for offline training for learning from a fixed set of failed and successful trajectories so do the modular neural Nets we built work for these robot problems well yes these videos are of the modular net solving each of these tasks but let's look in a little bit more detail about how well they work we're going to train the agent on a subset of the tasks and we're going to see how quickly it learns to solve them and then we're also going to evaluate the system on some other task that it didn't train on so for example if the agent never learned to pick up pick and place the block with the orange arm but it learned to pick and place other objects with the orange arm and it learned to pick and place the block with other arms then we're going to see how well it can combine those modules to solve the Unseen task if we train a separate agent a separate Network to solve each task separately we get only about 50% of the tasks solved if we train a bigger Network to try to solve all the tasks simultaneously we don't really do much better than that but if we use our modular algorithm we solve over 90% of the tasks within the same amount of data and then we can compose those modules in new ways to solve the other tasks uh about 80% of the time but the answer to this question is actually a little bit more nuanced than that these results require training the robot on almost all the tasks in The Benchmark if we train on a smaller subset of tasks we don't get either faster training nor compositional generalization and this is a direct conse quence of the fact that we're having the system learn the interfaces between modules on its own and that just requires lots of different training tasks so these results are in the easier setting where the agent gets to see a bunch of tasks all at the same time and one implication of that is that it gets to see how updates from one task affect another but we're interested in the lifelong setting so let's try to study that setting now the the robot is going to face tasks one after the next and we're going to allow it to store all the data that it wants from each task but we're not going to allow the robot to go back in time and interact with the environment from some task that it already solved in the past and we're also going to give the agent knowledge about exactly which task it is currently solving modular architectures give us several advantages for training in this lifelong setting we get to update individual mod modules targeted we get to increase the number of modules over time to handle more and more tasks and we get to compose modules and new ways to handle more tasks but as we just saw this comes at the cost of requiring many different task Durant training to solve the problems and the challenge is that in the lifelong setting the robot initially doesn't have many tasks it just has one and then two and three one way to get around this challenge all is to separate the learning process into two stages the agent will first assimilate the current task it'll try to use its compositional system to try to solve it and then depart from that system um to avoid making the assumption that the modules it has trained so far should work for the current task and then it's going to accommodate any data that it obtained from the current task into the set of learned modules to improve its compositional system so in the second stage the agent is going to try to build the best possible compositional system for future tasks and in the first stage the agent is going to try to it's going to leverage this compositional system but it's going to explore Beyond it to improve so how how can we do this one way we can do it is we'll train a separate Network for each new task we train and we're going to initialize it from the modules that we have trained so far far so we start on the first task and we just train normally via RL when we jump to the second task we're going to copy over any parameters for modules that we have already TR trained and then do reinforcement learning on top of that so the agent is going to repeat this process training copying over parameters for initialization and training a reinforcement learning policy on top of that early in the process the agent will have trained only a handful of these modules and so maybe we get a little bit of an advantage over training completely from scratch because these modules may have learned some useful internal representations but the overall policy that we're going to construct is going to be basically random because it has a bunch of untrained parameters but later down the sequence the agent will likely have trained all its modules and so it'll be able to compose those trained modules into a fully working policy and even if the policy doesn't work very well the agent will still do reinforcement learning on top of that to make sure that it can solve the task so this is the first stage then in the second stage the agent will take the modules that it trained in the current task and try to combine the knowledge in those modules with any knowledge that it has accumulated so far and the challenge is that the robot has to do it without going back to see those earlier tasks but while maintaining the knowledge that it has accumulated throughout the sequence so one way to enforce that the agent maintains this knowledge over time is to enforce a stricter requirement that it can still solve all the earlier tasks with this view we're asking the agent to maximize performance over a set of tasks without collecting any additional data for those tasks and the key Insight that allows us to solve this problem is that we can cast it as a multitask offline reinforcement learning problem the agent will treat the data that it collected while training on each task as a fixed data set and then it's going to to optimize a model in this case our modular model to solve all the actual tasks and once we see that we can use any off-the-shelf offline RL method applied to the modules that are used by the current task and over the data of the tasks that share some modules with the current task so to put everything together every time the agent faces one new task it's going to train a separate Network copying over the parameters from the previous tasks over to the new one and doing reinforcement learning on top of that collecting data that it'll use for the next stage and in the second stage it'll use offline RL over the modules that are used by the current task and across the data of tasks that also use those modules and then the agent is going to repeat this process throughout its lifetime for practical reasons we're actually going to evaluate these ideas on a simpler set of grit roll domains because training One agent sequentially on a robot Benchmark would take somewhere around three months so we're going to follow our compositional problem graph and construct 64 gritwell tasks and evidently this is a simplification of a robot setting primarily by making the state and action spaces discrete but we maintain the key property that the agent still has to discover the interfaces between modules autonomously we'll first look at the average reward across all the tasks as if we had trained them all simultaneously in order to get a picture of overall performance and what we see is that the modular agent learns substantially better than the agent that trains each task separately individually and we also see that we get zero shot generalization so before we even train on the task by combining the modules that we have learned we can solve the task partially and then via reinforcement learning we very rapidly increase performance and then we can look at the sequential nature of the problem so as the agent trains on more tasks we see the zero shot performance and we see that the agent is progressively getting better at generalizing compositionally it's accumulating knowledge that makes the modules work better and better over time just like we would hope for a lifelong learner so before moving to the next part of the talk I'm happy to take a quick pause here to see if there are any questions yes that's correct so we have discrete variations in in those parameters and then we choose one module for each of those choices of param yeah so we have a separate replay buffer for every task so yeah it continues to grow oh yeah okay so the first question was whether we um are varying parameters between one task and the next and choosing one module for each choice of parameter uh to which the answer is yes and then the second question is if the replay buffer continues to grow over time and the answer is also yes uh we only use actually 10% of that replay buffer so we don't use the whole thing but yeah it continues to grow over time good question um yes so that all is learned by maximizing rewards right because we're training the system on so many different combinations of things it has to learn some abstract Vector representation that can handle all the constraints of the environment that it sees so we're not enforcing those constraints they are learned by maximizing rewards does that answer your question and then I'm going to repeat the question because I forgot again uh so the question was whether we are enforcing some of the constraints like avoiding obstacles and so on in the in the modules or if that's learned by the composition of the modules yeah yeah so yeah the the the answer is this is this is learned over time uh so this is learned interactively by the agent it's not enforced in the modules yes so the question now is what do I mean by learning the interface so the interface is the output of one module should be com compatible with the input of the next module and so that's what I mean by the interface it's the output of each module module good question yeah one last question here great question so the question is do we really continue to learn new modules every time we face a new task uh and how do we choose like how how to avoid recreating the same module over and over again yeah so the answer is in this two two-stage process we create the new network for the current task and then once we merge it back with the rest of the knowledge we destroy that one right and so we're keeping overall we're keeping one single set of modules but every time we learn a new task we create a new version of that train it on that task and then delete it as we after we push it back into the rest of the modules yes so the question is how do we know which modules we are destroing so we have access to the full graph we know which modules are reused by which tasks okay so let's let's move on and when we we can take the rest of the questions uh at the end or offline okay so um we have seen that modularity and specifically functional modularity gives us a number of advantages for training lifelong robot Learners and we saw that it comes at the cost of requiring many different training tasks in order to learn those interfaces that that we just talked about there's one additional challenge with this formulation which is that reinforcement learning methods today are great at learning short Horizon policies but they struggle when faced with longer Horizon tasks like the household chores we started out with the key to enabling these longer Horizon behaviors is to go beyond the functional modularity and incorporate temporal modularity as well and one type of systems that embodies these two forms of modularity is Task and motion planning so T Systems solve robot Problems by breaking them down into two levels of search at the high level the task planner searches over the space of abstract High LEL actions like navigate to object pick object and then at the low level the motion planner determines how to execute each of those high high level actions with the actual robot so for folks with the reinforcement learning background this is conceptually similar to hierarchical RL with options where the task planner is analogous to the inter option policy and the motion planner is akin to the intra option policy but compared to hierarchical RL T introduces additional forms of structure which will make it a very promising candidate for lifelong robot learning being a little bit more concrete one way to solve a T problem is to use classical AI planning at the task level so we're going to use predicates to represent the state and operators to represent the high level actions and so in this kitchen example we might have predicates like Which object is on top of which and whether the fridge door is open and so on and we're going to have operators like navigate from A to B with preconditions that tell us where this action can be executed and effects that determine the transition from one abstract state to the next and then we might have another action like pick up the object and so on and then at the low level each of these operators is going to be tied to a um low-level motion planner with some parameters so for example for this action move from the counter to the fridge the planner might get the target XY coordinates as parameters and compute a Collis free path the overall system is built on top of a number of functional modules the perception module computes a predicate level state from the input the heris function takes these predicates and computes which are ranking over actions for which might be most useful and the task planner devises a sequence of abstract actions to solve the task at the low level the motion planner takes some parameters as input and devises a trajectory the transition function determines what the final state of that trajectory would be and a verification system checks whether that final State satisfies the constraint of the plan of the task and at the interface a parameter generator produces parameters for the motion planner given the task plan and this is the key piece that makes it possible to integrate task and motion planning and we're going to talk about it in a lot more depth later the system is then wrapped in a backtracking loop which tries many different samples and many different abstract plans until it comes up with one that works and then it executes it on the actual robot and the plan to solve a task is a sequence or composition of high level actions navigate to the fridge open the the fridge navigate to The Oatmeal pick the oatmeal making decisions in this modular way is precisely what makes Tam able to solve truly long Horizon problems so each of these highle actions is one temporal module and we're going to represent it as a classical planning operator augment it with a parameterized controller for executing the navigation and the picking and a parameter sampler to generate the parameters for that controller say the XY coordinates for the motion planner the key compositional capability that we gain from temp is that once we have the navigation and the picking and the placing actions we can solve multiple different tasks by combining these actions in different ways and because T Systems are both temporally and functionally modular we're going to be able to construct learning problems that are even smaller pieces of the problem so for example compared to the RL solution we don't need a motion plan to for the entirety of a task we can compute motion plans separately for each part of the task and conversely we can separate each abstract action into multiple learning t task say learn the controller learn the sampler learn the Dynamics function so we end up with a collection of modules Each of which is a mostly isolated machine learning problem and we have seen that that makes lifelong training fundamentally easier these gains evidently do come at a cost the reinforcement learning method that we talked about before was a fairly General machine learning solution we could apply it to other domains like dialogue systems autonomous driving medical treatment planning essentially domains where we might represent stages in a processing Pipeline with neuron Nets and where we might want to compose the neural Nets in different ways to solve different Problems by committing to Tamp as a base system we sacrifice that generality in favor of solving robot problems that users might actually care about as a starting point let's focus on the problem of learning parameters Samplers because that's the key piece that makes it possible to do this form of two Lev search and it also has some interesting properties as a machine learning problem that we'll talk about sampling based planning hinges on the idea that when the robot selects the high level action to execute it doesn't know exactly how to execute that action but it has a motion controller with some parameters a sampler that can generate parameters for the controller and a verification system that checks whether the controller succeeds for example if we have this robot here and we wanted to push the box into the screen it has to execute three actions navigate to The Box grasp the box and push let's focus on the navigation for a moment the parameters for that are the Delta X Delta y with respect to the center of the Box the sampling based planner is going to try multiple versions of these XY parameters on until it comes up with a pair that comes within reach of the box and that avoids Collision but what makes this problem really hard is that the rejection sampling has to be executed in tandem with the backtracking search because later in the plan the robot might realize that it can't push the Box in from the position it chose so it has to backtrack until it finds a solution that enables it to push the box in the efficiency of this form of planner depends heavily on the quality of The Proposal distribution the sampler if the distribution is too broad the planner is going to have to reject many samples throughout the process if it's too narrow it wouldn't generalize to situations where maybe there are unexpected obstacles so the goal of sampler learning is to learn a distribution that covers the entire space of in this case XY positions that allow us to push the BX into the screen so let's consider the following setting we start out with a working T system with a sampler that works but is pretty low quality like a uniform distribution the agent is going to face tasks one after the next and every time it faces a new task it's going to use its current Samplers to solve the task and then it's going to retrain its Samplers with any good parameters that it tried the key notion here is that the rejection sampler allows us to obtain a data set of good samples given a poor sampler and then the agent is going to repeat this process throughout the sequence of time uh of tasks learning new Samplers after trying to solve the task with it previous Samplers there are three Key properties of this learning problem that I want to highlight first the agent is faced with a natural TR train test Loop it is evaluated first on a problem that it had never seen before and then it uses the data it collects from that problem to update its model to be used in future problems that it has also never seen before the second is that the agent is going to be in charge of learning probability distributions and not classifiers or regressors and these distributions are going to be used in conjunction with the rejection sampling and third the agent is in charge of learning a set of these distributions and in particular these distributions might be some of them might be very similar and some of them might be quite diverse so if we look at the specific example of navigation it's very similar to navigate toward a chair or toward a coffee table but navigating toward door is fundamentally different so how do we deal with this last problem from a model moding perspective do we learn a separate sampler for each type of object one for the chair one for the coffee table and so on or do we train one single distribution across all the object types if we have lots of data for every type of object we almost trivially want to learn a specialized distribution for each of them but in the lifelong setting the agent will face new objects continually so it won't have lots of data for new objects and it might benefit from using data of similar objects to train its distribution the key is that we don't need our distribution to generate only good samples we don't need an exact model of the data because we have the rejection sampling all we need is to generate mostly good samples and to generate as many good samples as possible so what we can do is have the planner sometimes times draw samples from the generic distribution sometimes draw samples from the specialized distribution and let the T system figure out which are good so we're going to train one distribution over all the data across all types one distribution specialized to the particular object type we're interested in and so each of these are going to be in our work diffusion models but we can use any type of generative model that we want and then we're going to draw samples from the mixture distribution and so the missing piece is how do we determine the weights of the mixture distribution and the key once again is that we want our system to generate as many good samples as possible so we want to assign higher weight to the distribution that generates better samples but this is a chicken and neck problem we need to know the quality of the samples to generate to give us the weights of the mixture and we need the mixture components to generate samples for the T system to tell tell us which are better the solution is to bypass the T system Al together and use some proxy to measure how good the samples are and one useful proxy is auxiliary tasks so the high level idea is the is that we want to train our distribution and simultaneously train a supervised auxiliary model that is trained on the same data and that uses the same internal representation the key for these auxiliary tasks to be useful is that they should the labels for those tasks should be quantities that are related to the samples that we want our system to generate and those should be quantities that we can compute efficiently without going through the T system so what we'll do is every time we get a new state we're going to compute the ground truth auxiliary label we're going to predict the auxiliary label with our learned model and we're going to measure how close these two are and we're going to use that as the mixture weight intuitively if the agent can generate can predict a good uh can predict well the the auxiliary label in this state it means that it has trained in sufficiently similar states that it can generalize well and so we would expect the distribution to also generalize well in this new state so we want to assign higher probability to that that distribution so without going into the precise detail of things um geometric signals are one family of signals that meet these these criteria because understanding the geometric consequences of a sample is essential toward understanding whether that is a a promising candidate sample so to put everything together we're going to start out with the this T system that works but uses lowquality distributions we're going to solve the new problems we face and we're going to store the data of the samples that lead to success in those tasks we're going to measure how well we can solve the new task and we're going to train the generic and specialized distributions and auxiliary models and we're going to use those to construct our mixture distribution in the next Loop and then we're going to repeat this throughout the lifetime of the agent so let's have a quick look at how the system works in practice the folks at Stanford released this great Benchmark called Behavior where this funny looking robot is in charge of solving a range of complex long Horizon household chores what we'll do is we'll have the robot run through these tasks one after the next collecting data and improving its Samplers over time so very briefly what we see is that as the agent progresses over over time the efficiency of it solving problems uh grows better and better as compared to the poor Samplers that it started out with so as the robot is deployed for longer in these Virtual Homes its capabilities for drawing good samples improve over time for example we might start out by asking the robot to tidy up our home office so it'll come up with this high level plan to go to these folders here and bring them to the cabinet out in the back and the red dots are samples from the distributions that it has been learning and then the robot is going to move on to a next task where it has to pick up these cans that are scattered around the room and it will Leverage The distribution for grasping folders which are somewhat similar but it'll still have to specialize to the cans and the robot will continue doing this over the sequence of tasks one after the next stepping back a bit we ended up with a system that leverages temporal and functional modularity to learn these highly complex and long Horizon behaviors in these videos the robot is making several thousand low-level decisions but the planner only has to make 10 or 20 abstract decisions and then refine them with a sampler that the robot has been learning over time this idea of having a robot robot that learns over time but starts out with a working system is going to transform the way we deploy robots but there is still a long way to go the sampler learning problem that we talked about learns requires a fixed system that already works and a set of skills that have been pre-computed in some Factory stage but a LIF learner should be able to learn new skills over time as they become necessary to handle new problems skills that we may not have thought about during design as robots are deployed more broadly the amount of data that we have of real robots is going to drow grow drastically and so we need to be able to leverage data from across many homes in a way that both maintains specialization to each user's preferences and that preserves the privacy of each user's data and in today's day and age we need to be able to Leverage The Power of large scale pre-training data but we need to come up with algorithms that allow us to do to do that in a way that the learned the pre-trained models can adapt over time after they are deployed and as I discussed in this recent block post modularity is going to be the key that enables these capabilities of course all this work would not have possible without the help of my fantastic collaborators thank you and I would be happy to take any more questions all right thank you Dr mandes I think this is a wonderful talk uh at the beginning of the talk as I was reading a bio I was very curious how you put all these pieces together reinforcement learning planning and transfer learning and Robotics but yeah your talk very clearly illustrates how it works and like these pieces feel very necessary and yeah it's amazing talk thank you um yeah we can take some questions I think it's almost one but we started late so maybe we can uh take maybe five or 10 minutes of questions if you have any feel free to yes I can uh I'll turn on the voice lift system so anyone can feel free to speak up and the zoom will also capture the voice and folks on Zoom feel free to um also just unmute and ask your questions was there a way that was there a way that the earlier talk like is into the later half or are they kind of like different on a different like level in the hierarchy hopefully folks can hear me I turned off my mic just so we don't have multiple mics going on um they tie together in terms of the motivation of using functional and temporal modularity but not in terms of the approaches the approaches are quite different um but the idea that we can break apart a system to learn each of the pieces separate which is what makes it possible to do lifel learning uh is the the same one that motivated both of those ideas thanks again for the great talk so I have a question about this long life long like planning phase like long planning so uh if if the robot fails a task how how your approach uh to handle like these kind of failures for example you have this you show this like navigation and pick and place and some manipulation task but how about when robot the grass like the object just drop off from the river yeah so let me so the idea of task and motion B is that like the most common scenario is that you try a grasp and it doesn't work uh and so the idea is to backtrack and so here we're making the assumption that we have a simulator that tells us whether the graphs will succeed or not and so what we can do is we try the grasp and if it doesn't work we try another grasp and if it doesn't work we try another grasp and eventually at some point we decide well let's backtrack maybe the problem is that I am in the robot is in the wrong position so it should backtrack come up with a new position to navigate and then retry the grasping so if you had to do this in the real world if you don't have a perfect simulator that tells you whether things will work or not then I guess you would have to do replanning um but that will be costly so so hopefully you have useful simulators they don't have to be perfect but you want to minimize how many times you have to replant on the I see uh so I saw like the your work is on this lifelong robot learning if you consider any like you know online or on the Fly adaptation uh in can you clarify in what sense so generally like the the the like the great work generally shows how the robot accumulate L knowledge over time uh but you know uh I mean by online adaptation better the robot can sort of like adapt behaviors B on learning but show oh interesting uh no we have not but that that sounds like it's it's very connected and definitely something that that we should consider um but my work has not done that I think it's this idea of having a robot that can it's basically the idea okay you don't need lots of data sometimes on the file you have to figure out how do I solve this new problem right now um so that sort of adaptability is is very relevant to lifelong training but we haven't done that in our work okay cool question uh yeah let question so it's about ear part of the so when you have this like model learning doesn't require the tasks need to be like sufficiently distinct for example like what happens if your tasks like some of the tasks are like exact copy of other tasks but we don't know it that's a great question um that will have some implications because for example if if you're learning a module and it sees oh all these tasks require the exact same thing then all the modules will somehow try to overfit to that now if you have only a couple then that shouldn't matter because you'll learn two separate modules they'll both do the same thing but because you have other modules that do other things the previous modules in the chain and the later modules in the chain will still have to learn to work with all those different modules but if you have um many of these problems like if you have four modules that are the same and one that is different you're probably going to end up specializing all the rest of the chain of modules to these four things that are the same that's a good question yeah I I'm asking because you know it's like uh it's in the artificial setup it's like easy to come design tasks that are like right but I think learning like for your ultimate goal right some task task like you know exactly the same than they T they do before but it's like we don't know it just because they can yeah no that's a good question so and and yeah so I think I mentioned this at some point we we are assuming that somebody tells us what task we're facing but but in practice you know the robot might have to do not exactly the same thing twice but very similar things like maybe it's in some different state or something like that um so we do need mechanism I mean for example in the Tam setting um you know you know exactly which modules you're using all the time the Tam system system decides which modules to use it's not like it's brought upon the system by the environment the the system determines which modules it has to use and so in that sense you would kind of sidestep that problem that's that's a good time to think about there are a couple of question was an ear question also from ear part of the question so also how does the robot determine if a is different or the same it's probably related to my question when does it retrain the module or just reuse it yeah try to go to so the idea is that every time the agent faces one new task it's going to have two stages in the first stage it trains new modules and in the second stage it pushes all the knowledge that it just acquired back into the shared set of modules and then scratches the new ones right so we're not really growing this or I mean we grow over time if we need new modules but if we copied over a module we're not going to keep two copies of the market we're going to consolidate them in this accommodation we're going to consolidate everything together into one single repository of maret yeah and there was a question related to this is like if we save the data in the modules as new tasks come one the history incre size and I mean so one I guess one dend them to that answer is that like data storage is not too expensive uh you get to store lots and lots and lots of data I think for people that have worked in the lifelong or continual setting um one of the one of the challenges that really makes it hard to train when you're accumulating more and more data is that training becomes slower and slower right you have a growing data set then every time you have to go back and retrain it gets slower um the fact that our system only does that kind of in a short stage so most of the time it's just exploring on the new task and then it's going to push back knowledge in a shorter like post post training stage um I think that handles some of that difficulty thank you um in the first half of the talk you sort of enforce modularity where you say that these problems should be we should enforce modularity will help us have these modules have you thought all about ways that this modularity could become emergent through the actual learning process like neural architecture search or pruning or with Evolution upwards that yeah absolutely so so we do have some some work before this uh on the supervised case where we sort of do the process of determining the modularity autonomously so the system searches for um it's it's very similar to neural architecture search uh so the system determines what is kind of the modular structure of the task and then what right then trains the modules that it discovered and so on so yeah for sure I think that's a very important um important line of work um I think in robotics we know a lot about the structure of the robot problems that we're facing and so like just as a learning problem I think it's very interesting I think for robotics we can get away with enforcing the modularity most of the time y about like what kind of um State spaces were like the inputs here did you train these on like visual inputs or were they on a like a joint level inputs yeah we get the joint positions and the positions of the objects and so on so it's like a n it's a 60 something dimensional Vector that encodes a bunch of information about the positions and joints and velocities and so on yeah good question yeah I was curious to know how we can apply some safety constraint in um task learning or even skill learning because an important concern in U not just Lifeline Learning System but any kind of adaptive system is that they will have uh less or sometimes zero supervision from humans so it is really important to make the learning safe it's not just about learning how to do a task also it's about how to do that task safe yeah awesome question I so I think that that's there's sort of a two- C answer in the reinforcement learning case there's a bunch of literature on like safety and exploration and so on and trying to enforce safety constraints um which is relevant um I think think still reinforcement learning like in the Wild on its own is not the safest thing in the world so probably don't do that if you have an actual robot running around the house um or like an autonomous vehicle driving around the street so don't do that but the I think one key idea here is that modularity makes it a lot easier to enforce safety because now you can encapsulate your safety systems at each level right so you don't need the visual perception system to be safe like that's fine you need the decision making and the control at the low level of the robot to be safe and then you can impose safety constraints at very granular levels and so in robotics for example you you would impose like obstacle avoidance and that would be a rapper thing that goes on all the time and tries to avoid obstacles all the time so I think the key thing is modularity allows you to bake in safety easier because you get to do it at the modular level um we haven't explicitly thought about that or tried it but I think I think it's a promising direction to think about that great thanks there's no more question then let's give Dr MZ another run'll see you next week