Can This Robot Be My Personal Assistant

[00:00:00]I've always dreamt of having a robot and in 2026 it feels like that dream is a little bit closer all of the time.

[00:00:09]However, very recently it got much closer for me because on my desk is this.

[00:00:18]This is a Reachy Mini. Now, Reachy Mini is a super cute robot from Pollen Robotics and Hugging Face and it's not just like a product toy robot that you buy. It's a straight-up development platform that you can do anything you want with it.

[00:00:40]But just owning the robot wasn't really the dream. So, I set out to connect Reachy Mini to Hermes and give myself a true robot personal assistant. And because I need to justify having a robot like this, I set out on a side quest as well. That side quest is to build a spelling test app to teach my kids spelling because let's face it, learning spelling from a cute little robot like this seems way more fun than practicing it with mom and dad. But before I even got here though, we needed to build Reachy because this thing comes unassembled. Now, Reachy wasn't an easy build but it also wasn't a difficult build. It probably took an hour and 45 minutes and it could have taken much less if I was doing it by myself. This was a really great project to do with my kids. They got to ask a lot of questions and learn what all of the different component connectors were. We got to learn how to use the screwdrivers and how to connect things. This isn't their first electronics project but it definitely was their most involved. And when we finally got Reachy connected and turned on, there was moments of real genuine joy from them.

[00:01:57]>> Get behind him. He needs space.

[00:02:00]>> And privacy.

[00:02:02]>> Hi Richie.

[00:02:04]>> Welcome to the city, Marshmallow.

[00:02:07]>> Yo, what up, Marshmallow?

[00:02:08]>> So, now that Richie is built, it's time to get working on our custom software.

[00:02:15]Because we have a lot of things I would like to do with this. Now, the community-based apps are great, but I had bigger plan. So, to make Richie a true personal assistant and claim success here, I need this robot to do a few things. I need to, one, connect it to my Hermes. I need to get wake word detection, so that way I can talk to it directly starting out of nowhere. I need to have conversation going back and forth. And as far as the personal assistant tasks go, I would love for this thing to have a memory. I would like for it to be able to do tool calling. I would like for it to do simple coding tasks, be able to use my Home Assistant. And I would love for this to be able to do personal assistant tasks like simply reading my calendar. A lot of those are very possible with Hermes, but many of these things are going to require true custom software.

[00:03:10]Now, to give you some background, Richie comes in two flavors. There is the light version, which is what we have here, which requires the brains of it to be running on a computer. I have a Mac mini right there that is perfect for running long-running software on, so I'm going to be running my Richie mini software off of this computer, and Richie's going to be communicating over Wi-Fi. There's also the full-on version of the Richie mini that has a built-in Raspberry Pi.

[00:03:37]However, I think the light version, if you have a long-running computer, makes a ton of sense, right? So, what did I do? Well, I started off with a fork of the existing conversation app. From there, I used the prompt in the docs to really get started. I then replaced all of the calls to OpenAI with calls to My Hermes, which was really simple because Hermes has a great API. And this actually gave me a ton of features out of the box considering I use My Hermes for a ton of stuff already. So just simply by connecting it to Hermes, I was able to get memory via Hancho, tool calling of all the tools that my various profiles have set up, coding tasks so it can actually do coding on any of my machines since they're all connected via SSH. So simply getting those things working was super easy, but the tough stuff here was actually getting the wake word timing correct as well as conversation timing. Basically, the Richie software has voice activity detection. It decides when speech is actually starting and ending based on the audio level that's coming in. If the voice activity detection detects speech or silence, it then starts to transcribe via the local SST, the speech to text that's done being Whisper and it's happening on my Mac mini. From there, it looks to see if the word "Hey Marshmallow" has been uttered.

[00:05:03]Marshmallow is the name of this robot.

[00:05:05]My kids named it because just look at the guy.

[00:05:11]From then, Richie would then acknowledge the conversation and then start conversation mode. Richie basically just listens for the next sentence and then after 1.1 seconds of silence, which by all means is a value that I am still tweaking and I've had to tweak based on the way kids speak to this thing, the way I speak to this thing. It's taken a lot of trial and error here. It then transcribes that text locally. That text is basically sent to Hermes in a prompt.

[00:05:40]Hermes does its thing, returns the prompt, then our software generates text to speech using Cocoro using MLX on the Mac, which actually does take a bit of time and one of the things that I'm looking to improve upon the general speed and latency in conversation with this thing. We then send that audio to the robot and the robot outputs it directly so we can hear it. It then begins listening for the next prompt for me. So what you can see is it's basically just a conversational prompt.

[00:06:11]Now we also have some stop word detection where we say go to sleep and Reggie will stop the conversation.

[00:06:17]Again, this is still things that I'm tweaking and I set out to create a whole host of profiling tools so that I could understand where the latency is. Now one of the reasons why Hermes is less good for this is because by default a Hermes agents have a ton of additional tokens in here so the responses take a long time because we're sending a lot of information in each prompt rather than quick little fast prompts. It also depends of course which model you're using to have it respond to. No, I just have this connected via my Codex subscription so it is using some chunky models rather than some highly tuned fast models. That's something that I will be experimenting with more in the future. So, did this work? Well, let's start a wake listening and we're going to see if all of these things work and if we can check them off our list. I have Marshmallow here. We have wake word listening. We have conversation and we're going to first give this a try with memory and see if Marshmallow can understand. By the way, this is the custom UI that I've been building. Hey Marshmallow.

[00:07:26]>> Hey, I'm listening.

[00:07:30]>> What do you know about me?

[00:07:36]>> You're Scott.

[00:07:41]You like concise, direct technical progress, live checks, and family-friendly Reiki mini apps.

[00:07:51]You also prefer short robot prompts, and you've got two kids who may use me for spelling practice.

[00:07:56]I am listening.

[00:08:00]>> Go to sleep.

[00:08:01]>> Okay, I'll listen for hey Marshmallow.

[00:08:05]>> So, you can see that the conversation definitely works. Now, the latency is certainly there. It's something that I'm working on improving. Okay, now let's try some of the other things we had in mind. Now, I have several different profiles in Hermes that all have different abilities. And the Marshmallow profile is kid-friendly and kid-safe.

[00:08:27]However, I have a default profile that has access to things like my calendar, as well as can do coding. So, we're going to start a new conversation.

[00:08:36]What's on my calendar for today?

[00:08:44]>> Jury duty all day. Drop off from 8:00 0 to 8:30 a.m. Release phases tomorrow from 6:00 0 to 7:00 0 p.m.

[00:08:53]>> Change the color of the Marshmallow conversation app background to be pink.

[00:09:04]Now, the coding tasks may take just a second here, since it's going off and I don't think this is even inside of the project. So, it's probably going to have to go look for the project on my machine and then change those files.

[00:09:16]>> Done. The Marshmallow conversation app now has a pink background, and I verified it live at port 7860. It looks good.

[00:09:24]>> Yeah, it looks good.

[00:09:26]Marshmallow, uh can you turn all of the lights off, please?

[00:09:44]>> All reachable lights are off.

[00:09:45]>> So, now that I have the personal assistant of my dreams, it's time to get working on the hard part, which is the spelling test. Now, the spelling test has a number of features. It needs to ingest the spelling list of words. It needs to use them in a sentence. It needs to listen and respond and even give feedback if the user has gotten the spelling incorrect.

[00:10:13]Now, there's a ton of small little problems here, and it's time to get to work. So, after grinding on our spelling test for quite a while, we have something that can work that allows us to even save and restore various spelling lists. Now is the moment of truth. We're going to be giving the spelling test to my 9-year-old and see how he does.

[00:10:34]>> J U M P >> The word is jump. It is spelled J U M P.

[00:10:44]Try spelling it with me.

[00:10:46]>> I'm so mad.

[00:10:48]>> Okay, so the reviews were not glowing. He was a harsh customer, and he didn't like it.

[00:10:53]This is certainly not going to help him practice spelling, but I have some ideas on how to fix this. What if, instead of having a back and forth that took so long, we pre-rendered all of the audio, including pre-rendering any of the helpful text, any of the words that's saying you almost got it, and pre-rendered Reachy mini saying the word. That way, it's going to feel snappier back and forth. And what if we also output the letters on the screen as he was saying it? But most importantly, what if it actually worked? Because, as you saw, and honestly, what you didn't see was him spelling the words correctly over and over again and the robot consistently getting them wrong.

[00:11:37]Specifically because the window for when he was supposed to talk wasn't clear.

[00:11:42]So, I'm going to get to work on that and we're going to see if we can make my son a fan of this app. So, because of this first initial failed test, what I actually ended up doing was setting the Richie Mini in front of a computer, having that computer take the spelling test on the Richie Mini back and forth with audio and microphones, and then that way the system was able to self-heal and improve the latency using Open AI models to run the test, check the debugging information, and then fix it, test it, reiterate until now we have an app that is, I got to say, working pretty well.

[00:12:20]>> Your word is robot. The teacher asked us to spell robot.

[00:12:24]>> Um r o b o t >> Nice work. You spelled it carefully.

[00:12:36]Your word is eat. The teacher asked us to spell eat.

[00:12:39]>> e a t >> Nice work. You spelled it carefully.

[00:12:49]Your word is jump. The class practiced the word jump today.

[00:12:54]>> j u m p >> Nice work. You spelled it carefully.

[00:13:00]>> So, I think this was a great success.

[00:13:02]The robot is now my personal assistant.

[00:13:05]It's taking spelling tests with our kids and it's only going to get better from here because I still haven't really implemented that much with the camera.

[00:13:15]The next step on the docket is to use the camera to teach my son and I how to play Warhammer 40K. We're going to use the camera and the LLM that this is talking to to discuss both strategy as well as what to do next so that way it can even identify pieces on the battlefield. That is a big project and the one that I'm going to be diving into next. That said, I'm a huge fan of this little robot. It's a really cool open project that you can do all kinds of stuff with and I got to say, I think I just made a new friend.