What we're working on - Menlo Community Talk

[00:00:02]Hi everyone, welcome back to another Menlo community call. Um it's very good to see everyone here. Uh I'm Mimi and I'll be your host today and um whether you've been with us since the very first call or you're just joining for the first time. Uh thanks so much for hanging out with us today. Um and as always, if you can't stay for the whole stream or um you're just catching us later, no worries at all. Every stream gets uploaded to YouTube afterwards, so you can always come back and watch it whenever it works for you. Um, and if you're watching this live, go and say hi to us in the chat. Uh, we're very friendly here, I promise. And, uh, we'll be keeping an eye out on the chat throughout the stream. So, if you have any questions as we go, uh, please let us know and we promise we'll do our best to answer all of them.

[00:00:51]So, uh, for anyone who's new here, here's a quick idea of what these community calls are about. So, every week we like to sort of pull back the curtain a little bit and show you what we've been working on. And that usually means walking through what we built, what we potentially broke, uh, what we learned, and where we're headed next.

[00:01:14]And today, here's what the lineup looks like. Uh, you may notice we don't have our usual hardware team here. And that's because they're busy cooking up something very special behind the scenes. And we will hopefully be able to share uh more news uh with you later on.

[00:01:31]But uh instead this week, we'll share a few broader projects happening across the entire company. And uh we've still got uh plenty to show you. And we'll start with software team. Then we'll dive into what our brain team has been working on. Um move over to robot systems. Uh share some fulfillment updates. and we'll wrap things up with careers.

[00:01:52]And uh as usual, if something catches your attention while we're talking again, just don't wait until the end and just drop your questions into the chat and we'll answer as many as we can.

[00:02:02]Okay, so let's just uh get straight into it and I'll pass on to our software team.

[00:02:17]Hey um good morning, good afternoon everyone. Uh um today I'm I'm going to share with you our progress in building our platform client app. Uh but first uh for anyone new here uh and to refresh some of your memory uh about what I shared last week um we proposed a um an security architectures for our robot called robocalling or in short. Um in this architectures um the re the server main server or any other relay server u I got a server it cannot fork any uh cannot uh uh fork or modify or like create any um unverified command to control robot. So it ensure that even if the server is compromised uh the attacker cannot control or cannot move the robot. Uh and we build that into our client app uh which I we uh happily show you now.

[00:03:29]Okay. So this is a desktop version of it. Um, we create um simulated uh CPU uh the the one that is running on the ASMO um hardware. Uh right now it apply uh so let me let me start it.

[00:03:53]Okay. So you can see the the grid dot here meaning that uh it it is online and uh we can start connecting to it.

[00:04:03]Um this keep in mind that this this is a uh uh CPU emulator. It it doesn't have a body. So that that's why you're seeing uh just a simple dot here showing uh the green one is where you think that it's facing and the blue one is like it chase. Um this one this one you pair with this device and the tool we communicating using our robocalling protocol. Uh yeah we we are going to integrate it hopefully with the real robot soon maybe tomorrow uh maybe uh the upcoming week.

[00:04:48]Um on the desktop you can also have you can also see other setting as well which we are working on. Um you can also back up your recovery key uh which you can later on um uh import into other device or um you uh later on you can also generate some uh key as well here to to start controlling the robot from the SDK.

[00:05:18]Uh let's see the the mobile app. This is very much similar to the desktop one that you can see here.

[00:05:39]Can also switch the team. You know, it's not working very well yet. Yeah. Uh and let me also show with you some of the the thing that my teammate did is connected the PS5 controllers to the network client and start controlling the robot and uh yeah I think that is uh for me to share with you Today um we are working on a lot of thing behind the scene and uh yeah hopefully next week we we can share with you a lot more fun stuff.

[00:06:28]Yeah, thank you.

[00:06:31]Back to you Mimi.

[00:06:35]>> I think there's a question from Nicole actually. Do you want to address it? Uh >> yeah. Um as in asymmetric encryption. Um, yeah, let me uh let me open up the link again. Um, I don't know why my my laptop is lagging.

[00:07:12]Um, I think your laptop has been working very hard.

[00:07:24]>> Yeah. So it it is running a lot of simulation. Um yeah. Um so so there's some um spec you can um you can look at and the link is share on discord. Uh I I will try to find the link right now and pass it here. Um we are first I need to clarify that we are not do doing um encryption yet. Um this is not part of the robocalling uh spec. So so the command when it's sending sending between the robot and the client not encrypted but the protocol we are using we ensure that like um we use the key to sign another key. So that is certify the other key. It's not like encrypt the content of the the the data if it makes sense. Um we do use some encryption to back up the key to our server with pass.

[00:08:26]Uh yeah uh and I hope that that enter uh answer your your question.

[00:08:41]Yeah, we we use some algorithm uh I think uh ago uh ID or something like that. Yeah. To to encrypt the backup uh block of your private key to the server with a a strong passph.

[00:09:07]Yeah. So the motiv um it allows you to um so it allow you if you um lost your device, you can still enter your password and you can still continue using uh your account in another device.

[00:09:38]I think there's another request um whether we want to talk a little bit more about the ICL spec and maybe the community has um any feedback or maybe ideas. Is there anything you want to share James?

[00:09:57]>> Oh, I can say time.

[00:10:00]>> Yeah. So, um please reach out to us on Discord. Uh we would love to discuss with you in depth about uh our um security uh proposal. um would would love to really love to the community to shed it.

[00:10:20]Uh yeah, I know that a lot of people in our community is uh like have been working with it in a while uh in in in the field for in the security or in the hardware field and we would love to hear a lot more from them.

[00:10:40]Uh yeah, so another question. So we we um we still have an account login system.

[00:10:49]Um this is for you to address the organization um uh setting up a permission control the the normal um uh robot access controls uh and like organization and team stuff.

[00:11:06]And it's also allow us to um verify that you you um basically store the um all the key backup and everything and also allow the uh hosting of the lit so that the robot and the device can connect to each other. So a login is still um mandatory.

[00:11:34]We could make it optional in the future but I think for version one uh we still keep it.

[00:11:42]Uh would they support multi multiffactor authentication? Um for for login uh for login right now we don't have it yet but it it can be enforced for key one. Uh I'm not sure if you for for for the key one we might also do that so that like it uh you need to enter your uh biometric or your 2B to be able to download the um recovery uh uh the recovery backup on the server.

[00:12:23]Yeah. But but it's still very early. We are still um refining the flow so that it balance between security and user experience.

[00:12:38]Okay, I think uh that's uh all the questions for now, but again, please just feel free to drop uh more comments, feedback, uh ideas, anything into the chat. And in the meantime, we'll move on to Ishnit and we'll just come back to more questions at the end if you have any. Thanks, Jane.

[00:12:59]>> Thanks, Hi, good morning everyone. Um last week I think Jackie spent some time talking about what we are trying to build with the brain team and what we're thinking about along the lines of a large physics model as the eventual goal and this week we actually did some experiments to prove certain hypothesis we had in the initial phase and we also consolidated all our learnings into a paper and I'll be walking through the main learnings and the motivation behind what the experiments we conducted in this paper are and we hope to release it by the end of this week so that you know the community can read through paper, look at the open source repo and, you know, build on top of it. Give us more ideas, tell us if we're doing anything wrong or not. So, the motivation for, you know, why we actually even want to build a multimodal embedding model for robots is because, you know, like when you pick up a cup or, you know, when you're filling a glass, you just don't rely on vision and you just don't rely on like pixels.

[00:13:56]You rely on, you know, the weight of the glass. You also rely on the sound as the glass is being filled up. And so this actually shows us that the different senses are actually entangled in a way and you need to rely on all these sensors to actually be able to do a realistic task in the real world. And vision is just one sense among several and it's not the most important one for manipulation. And now if you think about it, how do every you know other robot learning system work? They usually run a camera through a big vision model and then they get some form of representation and then you know almost as an afterthought they staple the other sensors. So they have some audio which is encoded. They have some force which is encoded and they also have propriception data which is encoded. But the core understanding of the module in you know most of the systems nowadays is just pixels. And so when the system relies too much on pixels, everything you know that a robot does downstream is basically limited by what its encoder captured. And so if information about force never makes it into the encoder summary, then you know no policy, no world model, no predictor can actually reason well and get the robot to do something, you know, really important or actually get it to fulfill the task. And so we actually even verified this in one of our experiments that you know if we just encode vision can we actually get anything about the force in the um after the encoder and you know we actually cannot. So if we just encode vision you actually don't get any information about force. So vision only encoders like fundamentally and structurally blind to one of the most important signals in manipulation. And so what a motivation right now is that you know we want to build an encoder that takes in multiple modalities. It takes in vision. It takes in prop perception, it takes in audio, it takes in force and then it fuses them into some form of like single shared summary and that makes the foundation actually reusable for whatever predictor whatever DIT vit backbone you want to use later on to actually reason about the task and then you know eventually you can decode it into action chunks and the robot can actually execute it. So I think the main things um we're sort of contributing in this paper right now is that one you know you know vision has vit or clip as the encoder language as LLMs but robotics still sort of uses these separate encoders to actually encode all these different sensor modalities. What we are saying is that you know we have to build a multimodal encoder from the robot's perspective first and also because our robot in this case is you know open we control all the sensors we can actually understand the robot better from a hardware perspective as well. what sensors are the most cheap, what sensors are the most accessible to put on a robot and then use all these sensors and use all the information that the robot exposes and you know package in into a very good very compressed latent and that will actually help a lot of the downstream tasks to actually reason about whatever the robot feels, whatever the physics the robot experiences and reason about these tasks well. And so what we plan on what we try to do through this paper is we have a robot first multimodal encoder and we able to show that you know crossmodal fusion works. We also were able to show that you know the data set we use the encoder is actually embodiment agnostic and performs better than sort of specialist encoders and that you know vision only latence cannot actually recover the force that the camera cannot see and we also prove some downstream uses of the frozen encoder.

[00:17:20]So we were able to show that an encoder can actually be used for some safety scenarios as well as you know the basic principles of an encoder. Is it even able to reconstruct vision properly? Is it able to reconstruct robot state properly? All these things are also preserved. And so when we when we start with this um project we we have to think and ask ourselves a question you know why an encoder and not just a policy directly. So you know a policy is trained for only one task and an encoder is trained and you it can be reused by everything. So if the representation is good you can evaluate it cheaply with a small probe instead of retraining the whole robot policy. So the encoder is actually the main leverage point that will allow a lot of downstream tasks to come and flow very smoothly.

[00:18:05]And so why self-supervised right data has no labels at scale and it's really very hard to annotate multiple different modalities in the robot data sets. So you can't you know expect someone to annotate millions of frames of force vision audio and so ideally we want the encoder to teach itself from the raw streams and the way we can do this is you know with self-supervised and why not you know reconstruction or using autoenccoders. So one thing we realized with that is that with autoenccoders you're sort of memorizing every pixel and you can just sort of reconstruct it but you're not really understanding the reasoning behind it. And you know why not contrastive losses or you know just make um audio v video and force equal.

[00:18:50]When you do that you're just sort of taking the intersection point of all the different modalities and you actually lose a lot of information. And this is what a lot of um research is currently doing. They basically take audio, they make it into a specttogram, and they basically treat everything as almost vision. But by compressing it into a form factor before it's actually encoded, you lose a lot of raw value from that sensor itself.

[00:19:14]And we built specifically on Lijah, which basically has a simple mathematical trick um in its loss function called sigre that basically proves um it stops the model from cheating by collapsing everything to one point. And so this actually allows us to take advantage of the lipa and the whole japa architecture to prove a concept with the encoder. And so the key idea over here is you can see in this figure is to fuse wildly different sensors. We have to put them into some form of common language. So we just turn every single sensor into a small token where it has the value what it measured you know the pixel patch the joint angle force reading the audio what which is which sensor it came from and also where is it. So it's spatial embeddings and also when so we can actually encode each different sensor with its native sampling rate using some form of real time step. So the encoder also respects the time and so one neat trick in this is that instead of you know truncating all the sensors we sort of use a masking approach. So different robots may have different joints some may have a force sensor they may not have a force sensor.

[00:20:22]So instead of designing custom input for each robot, we use one fixed layout and just mask the other parts of a robot parts that the robot doesn't have. So it actually helps us to make the encoder feel like if there's a missing sensor, it's basically that it's masked. So the model is also robust to death sensors by construction. And one encoder can actually work for every robot. And so the main architecture we actually experimenting with is this cross attention fuser where the where where we take a image and we put it through a frozen a frozen pre-trained V or vision model and then all the tokens after that go into this perceiver architecture which is basically um it it will ask summary questions that read over the whole bag of tokens and then it distills it into a very compact latent and so you know there's a question why would you use a perceiver and not just a giant transformer to learn the attention across the latent for the perceiver. It doesn't blow up as you add more sensors or tokens because the summary is always in the same size no matter how many sensors are actually coming in which is what exactly lets the masking handle different robots cleanly. And an important detail for this is that at test time we feed the encoder vision only. So we hide motor and force and we are able to actually derive from that that the latent has actually learned good information about the force and the joints just from the pixels alone. So you know if you don't have those sensors if you just deploy with a camera but the latent is actually trained on multiple modalities even with a camera it's sort of able to mimic those other modalities as well.

[00:21:59]So the way we actually train is let me show it here.

[00:22:04]So this is um the main losses we use in our training objective and it's basically very simple where we just want to hide one modality and predict its meaning from the other two. So what we do is we can hide the force and predict what the force should be from widget and joints. We can hide vision and then we want to predict what the vision can be from the joints and the force.

[00:22:24]So this actually works because a shared multimodal encoder without this crossodal sort of losses actually does worse than single modality models because it just you know combines everything and concatenates everything together and the crossodal prediction is where it actually makes the fusion work and you know it makes the latent really compact, really compressed. It makes sure each modality sort of entangled with each other. And I can I'll talk about the results we actually conduct, the experiments we conducted and the results we got that actually prove some of our points here.

[00:23:01]Yep.

[00:23:03]So from the results we were able to prove that we can actually train one encoder for all robots. So if you look at this here, we can actually train an embodiment agnostic encoder and this is um where we are able to show that you know the encoder trained on all the embodiment actually outperforms the specialist encoder trained on all. So what we did for this experiment was we trained it on four different robot arms at once and we asked you know whether a specialist encoder will do as well whether a generalist encoder will do as well as the four specialist encoder train on each robot embodiment and we were able to show that actually the the generous the generalist robot is able to learn more cross embodiment and crossmodal behavior in general rather than a specialist encoder. And the second result we got was that um let me show this figure.

[00:24:03]>> Yeah, this was the main result we got as well is that we were able to actually take the vision only latent to predict false talk better than raw vision features on every robot that has a force sensor. And this means that you know fusion of the crossodal features actually is able to deliver where it should where the camera is blind and you know where the camera doesn't have information about the force but it's actually able to predict force and other modalities better.

[00:24:30]And then we also had some downstream experiments where we try to train try to do some form of safety detection on the encoder where we just wanted to know that you know if the robot state is not what the vision expects can the models crossodal prediction actually you know spike when it sees like invalid states and sees if the robot is not doing something safe and then we also did some basic decoding on whatever the encoder was able to train and we just wanted to see you know if a tiny decoder can recover joint state from the latent as well as we used um this diffusion model called pix nerd to just see if we can actually just decode back video from the latent as well and whether small generator can redraw the camera frame from the laten showing that you know the 256 dimension laten is actually quite um intelligent and it has actually retained a lot of information and so with this main experiment um you know one caveat is that okay it's little single time step. So it's not actually learning anything across the temporal phase. And what the next experiment we want to design is that we want to do it across the temporal space and um see if the model is actually able to learn you know in time not just across modality.

[00:25:46]And so what we will do for this is that because our our each sensor has um a different sampling rate and we can't just concatenate them to one sampling rate and then you know just fuse them together and put them into the perceiver. What we do is we basically associate each sensor stream with its real time stamp. So almost like a temporal embedding and then we can fuse the streams across time at their native rates. So we don't have to resample and we can use the same architecture. So you just basically switch the when part of the token. So you just associate a time with each sensor stream and this is the path from like a static encoder to a genuine world model. We're able to predict the future, reason about dynamics and you can condition on actions later on. And so this is what we'll be focusing on eventually. And then some limitations of a method so far and the experiments we are doing because this is still quite in its nent stage and we're trying to figure out what will sort of be the best in code architecture that will actually work on the real robot as well. So the main limitations is one it's you know currently it's single time step there's no temporary results yet but this is what we want to do next week and also it's correlational not causal. So it learns that the modalities co- occur but we can't still prove that you know it's cause and effect and so also the data set is quite it doesn't have a lot of variance and eventually what we have to do is once we are able to train a VA on the arms and the robot we are working with we'll be able to actually prove the encoders concept better because we'll be using the exact data we will be using to like deploy on our robot rather than just using a general data set which is available online. Yeah, if you have any questions, you can ask in the chat. But this was basically the conclusion for whatever we're training with Kepler encoder, which is basically a robot first multimodal encoder that will take different modalities and actually make the latent really really smart. So that whatever downstream task you have. So after you have the encoder, whatever predictor you want to train, we'll be actually able to reason very well because a lot of information is intelligently compressed in the latent itself. Yeah, that's all from my side.

[00:27:53]Thank you.

[00:27:55]Thanks aid. I think there's a question from uh Richard Franklin. Maybe you can share a bit about the about data.

[00:28:03]>> So the data we used is this this opensource data set called um RH20 which is basically a data set which has multiple modalities. So in that data set there's audio as well. There's force sensing data. There's robot propriception data as well. And we just wanted to prove we just wanted to quickly prove a concept with a multimodal encoder. So we just took a data set off the shelf.

[00:28:35]Okay, thanks.

[00:28:38]I think we can move on to uh Salem next.

[00:28:45]>> Um hello everyone. Uh I'm Sim. um one of the member of the technical stuff here at Menlo. I think this is one of my first uh community uh calls here. Uh so um today actually we want to go through a little bit of like um what we call the ASM manager which is a new form of like software that we are going to develop. As you guys know like we have a Raspberry Pi that is connected to a Lola controller which we call the robot processing unit the RPU. Um the RPU is basically that part that runs the locomotion policy that talks over CAN to the motors that does all the real time um inference and real-time control. Uh and the Raspberry Pi is basically the one that is connected to the network and can set up your uh overall computer like overall robot sorry. And um you know while we were like working with that you know uh we realized that would be great to have a almost like a router logging page where you can log into your robot um you know from time to time. It does need to be logging every time. Uh and then you can basically set up all the different services. You can set up you know uh security keys and um basically start and stop different services that running on a robot. And this is what we this is where the asimoth manager was born. It's basically very minimal uh control and conflict service that runs on the robot um ideally on the Raspberry Pi. Um this is how we want to going to do it. So how it actually works you know there are like lot of different like systemd services running on the robot.

[00:30:31]There is the system the service of the asmov firmware the smok API which is uh documented in the manual online and um you know we are basically trying to uh create um a CLI where you can basically check the different status of this. So you don't need to be logging into the uh RPU directly. you can actually be just using um everything on a Raspberry Pi and get the entire information about the configuration of your robot uh pair um pair CLI commands pretty much. Um so you don't have to touch the code, you don't have to um open different, you know, uh shell files or like bash files or or any like config files. You can literally just use this API uh this CLI sorry and you can basically start up your robot uh very simply um and how it actually looks and I think we should also discuss a little bit about this architecture um because um this is quite new. So as you guys know um again we have the RPU which is where the Simo firmware runs. The ESO firmware is a real time um uh real time application. It runs like on realtime Linux. Uh we have been verifying it uh over and over that it meets the timing requirements to perform the locomotion policy perform the communication over can. Hence we want to make sure that nothing that this device is basically not you know responsible for networking for like connecting connecting to different clients. We do want to separate this as this is the real-time piece and runs with very high priority on the entire system um on a different board. Uh that's why we basically disconnected this actually also electrically. Um the Raspberry Pi is the device that is responsible for like connecting to the network, running your individual applications, your web servers and everything. Um this uh the Raspberry Pi is also running the SMOF API. So the Simoff API is pretty much maybe you guys know from the Unity world. Um the SMOF API is how you can talk to the robot uh internals.

[00:32:50]You can basically ask for um states of the different motors, states of like temperature of sensors. You can p anything you want that is robot specific. And you can also run and control the robot through the Simov API.

[00:33:06]So when you when you would run this um on the cloud um you would technically just talk to the ESM API and your LifeKit SFU which stands for signal forwarding unit would run on the cloud.

[00:33:21]But you know as as we like develop things locally um and test things uh offline we actually thought about just moving the lifekit SFU also into the Raspberry Pi since the lifekit SFU is actually a very lightweight um program.

[00:33:37]It can run on a very small even microcontroller these days. So the lifekit team has done very good work there. So we actually pull the lifekit SFU into the Raspberry Pi um during the setup. So you can actually mimic how you would connect to our platform in future.

[00:33:57]So the Simoth API is then basically talking to the live SFU which then talks to your uh client application over web RTC and this can run anywhere you want.

[00:34:07]This can run on your computer. This can run um this can run on on your toaster technically. So um you know as you guys can see we we have like three services we need to set up um with the correct sequence. There's a live kit SFU, there is SMO API and the ESO firmware. And to set this up in the correct sequence, you need to run this like you need to first run as of firmware, then you need to run the Simoth API, then need SFU. And each of those components actually has a check that like a heartbeat check that it that it's neighboring component life SFU or the ESO firmware is alive. So there are checks all around to make sure like that the system is properly set up before you can actually operate it. Um so there is a heartbeat between these two guys.

[00:35:00]there is a heartbeat between these two guys. There is no heartbeat here because I think the client application can always go down. There is no there's no reason to uh basically kill the system and to actually set up all of these different uh services.

[00:35:17]There is the asimoth manager. The SMS of manager is basically this one component that I that I showed previously where you can it's almost like a router page.

[00:35:26]can log in to a different port just locally. Doesn't need to be on as long as you're on the same Wi-Fi, I guess.

[00:35:34]Um, and then you can basically make sure if all the components are um already running by default. Technically, when you turn on your robot, all these services, they should come up automatically themselves. Um, the SMOF manager is really a a one time when you buy the new robot to set this up the first time and same as a router, right?

[00:35:57]A router when you when you buy a Wi-Fi router, you set up it once and if if something goes wrong. You can go back to the router login page and change certain things. So the first time you buy a robot, these services will be offline.

[00:36:12]Only the Soft Manager will be online.

[00:36:14]You connect to it, you start the different services. All the different services you can individually test through the Soft Manager API, Soft Manager logging page. And once all is set up, you can basically um reboot your system and from there on everything will run automatically. Um so there is uh that's the idea behind this. So the the SMS of manager is really first time usage and if something goes wrong um you can check certain things as well as when you say okay I want to change my life kit access token um same as you want to change your Wi-Fi password you just log into the router page you enter the new data uh your new password your new secret key um and then basically uh you reboot the system and it should automatically pick it up Um, so the web UI, I think we need to do this little bit bigger next time, but the web UI is actually very simple. You can just see the different services and we kept it very simple. We kept it like uh we had made sure that the services itself have most of the logic and we don't have like a mesh of a lot of services. So there are just really three services that are running to operate the robot.

[00:37:32]um the locomotion policy connecting to the cloud all are just running to three services the lifeguard server the API and as firmware. So in a dashboard you can see um if they're running the state you can see if you want to auto start them in the beginning you can restart them you can stop them um you can set your password and uh this is almost like the router/config page actually looks like um so yeah I think this is pretty much it um I think we we going to launch this soon I think this will come online as well as open sourced. The team is just preparing us right now um cleaning up the codebase. So, and then we should be able to um open source this and uh and if anyone has uh is interested to make this more user friendly or better, feel free to just contribute to the open source repo when it comes up. Um yeah, thanks everyone.

[00:38:39]>> Thanks.

[00:38:41]Um, I think we can move on to Frank next. And I know that we have questions coming in. So maybe we'll just have a Q&A session for the whole team afterwards if that's okay. So please continue sending your questions.

[00:38:56]But in the meantime, we'll move on to Frank with fulfillment updates.

[00:39:03]Yeah, thank you Mimi and uh I'm Frank and it's pleasure to give update about our asmo fulfillment status. As I shared uh last time we have a vacation unit building this month and also target to deliver our first batch of the ESM in in August. Uh so far everything is changing and uh uh we are having daily confidence call with our contract manufacturing and the key supplier to make this happen and uh uh in the meantime and we are also working our uh contract manufacturing to on the uh uh delivery substance and the target is to provide uh more and more door-to-door service for our customers and uh also we are also working on the uh bone update and also manual update.

[00:39:50]data and uh once it's done and we uh publish it to the our website and then that will be easy for the uh these people that want to build the asmo by yourselves. So uh actually a lot of thing happened here and uh uh we are working to make as more and uh uh easy available for the whole community and uh uh I believe we have more update in the coming to the weeks for the uh as the building status and also the DDP service. Uh yeah that's all from my uh update from my side. Thank you.

[00:40:30]>> Thanks Frank. I think we can uh have Herman do our quick little promo for the new um career opportunities we have.

[00:40:42]Give me a second.

[00:40:44]>> Hi.

[00:40:44]>> Hi everyone.

[00:40:46]>> Hi everyone. Very nice to meet all of you today. So as usual we have lots of open roles. I think uh end of this week we'll be also pushing out a role for a world model researcher. So, I look forward to meeting each and every one of you and I look forward to speaking with you guys as well. Have a great have a great rest of the day.

[00:41:08]>> Okay, thank you Herman. Um, I think what I'll do now is I'll put my uh our entire team on stage and we'll just blast through all the questions we have. Uh, I know there's a lot to go through, so please bear with me.

[00:41:26]Um okay I think maybe we can start from the end actually.

[00:41:41]Yeah I can take this question. Um so for this actually the critique is well that the epsoly statistic is a distributional test and it almost sort of estimates the embedding distribution but I think um because sigreg is an anti-olapse regul regularizer and not the learning signal I think the temporal dynamics are meant to be learned by the predictive objective so whatever like you know your action condition forward prediction loss is at the end and so I think we try to sort of mitigate this with a design is because we apply cigarette per time step not to a time pull latent. So it computes the isotropy constraint on each time steps latent. So you actually never pull across time and each time steps marginal stays healthy while the sequence is free to vary. Yeah.

[00:42:39]>> I think hardware team isn't around. Um but I think for the by I see >> um Daniel I think I can take this on.

[00:42:56]So you can technically use a different uh gripper. I think the uh connection that it goes into the SOP V1 arm is like uh RS ser RS252 serial connection. So you can actually as long as it fits in mechanically um you need to write I think you need to write your own driver and that connecting to the um s of manager or as of API. So um that that that is possible and uh hopefully we that would also great if you can share this on discord later on. Um, but I think it all depends on like uh the um the gripper interface that goes into the um Esmo V1.

[00:43:52]>> Yeah. Uh Ishi, do you want to take this one or I think I I I mentioned in the in the YouTube comments. I think the we we definitely need to take a look at the data. I think this is the config 7 that has like a different set of either motors or or force feedback. Um and and and I think Franka might have been the one that had the least least data in the data set. So that's also one of the reasons, but it's definitely something that we will investigate. Yeah.

[00:45:04]Yeah. Um, thank you for the um, thing with the cross attention approach. Yeah, I think it's definitely needed to actually make sure the modalities learn um, about each other in the same latent.

[00:45:15]So I hope it also works on the real robot but we will be investigating this more and more as we scale up the experiments and actually move on to the temporal phase so we can actually do more model like work.

[00:45:40]Okay. Do I have any more questions?

[00:45:45]Um if not perhaps we can close this off for today and then we can continue our discussions on our usual channel discord. So I'll drop a link here so everybody can um join our discord and otherwise I think we can wrap up here for today right and um as always thank you so much for spending uh some part of your day with us. If you enjoyed the stream give it a like. uh subscribe uh wherever you're watching and follow us on X and Discord to you know keep you up with our uh updates and we'll be back again next week with more progress and probably a few more surprises along the way. So thanks everyone and uh we'll see you next week.