Fully Local AI Voice Assistant

[00:00:01]Around 3 years ago, so right here in YouTube, I recorded a video called death of a software engineer and a rise of a product engineer.

[00:00:10]You know, I was even back then I was to a huge degree certain of my beliefs, but unfortunately my beliefs turned out to be true way more than I anticipated. And here we are today, 3 years later, we are basically pressing enter, approve, or choose one of the these three options from your harness. That's what basically software engineering for many people has turned out to be.

[00:00:36]Now, I was trying to, as always, us being engineers, my whole career I was trying to obsolete myself from repetitive tasks so I can focus on actual problems or talk to customers, talk to my CS team, my marketing team, my designers, see whether I can actually help them instead of writing another React component or an API. So, as before and as I'm doing today, trying to optimize my workflow, I came to a conclusion that the future of programming, another prediction that I'm making, is fully vocal. It's speech-to-text and text-to-speech. And let me elaborate why, and then I'm going to show you how I will move forward from today. So, if we pay attention a little bit to the current conversations with AI, that is you sit, you generate a huge plan that you didn't read, then the the AI regenerates or updates that, then you also don't read that fully almost never.

[00:01:35]And then you approve something and then 20 minutes later, 30 minutes later, 3 days later, you have some app that in most cases doesn't do something that the way that you want it.

[00:01:46]The reason why that app doesn't do everything that you want it is because, well, first and foremost, again, you didn't read most of the things, and second, the stuff that agent is spitting out is meaningless implementation details. So, uh I think we're already past the time when we care about lines, at which line this function is going to be generated, what name of the module is going to be, because hopefully you have solved the technical implementation details, the code, the syntax using your harness already. So, simply put, if your agents are producing code that doesn't look like the code you wrote, you failed, you need to make a step back, and you need to either build your own harness or add the guardrails to your existing harness, that being skills or other markdown files or hooks. Whatever it is, you need to solve a problem of, "Hey, is my harness generating the code that looks exactly like how I want it?" If answer is yes, then essentially you're on the right track, and this video is going to help you. So, if we take that fact now and moving forwards, because that is certainly true for me, I no longer want to get the output from my agent talking about code, because I know that my guardrails will ensure that the code that gets generated is already at worst 95% what I want it.

[00:03:07]Of course, I'm going to review it. Of course, I'm going to ask other agents to review it, but let's just for the sake of this monologue say that that's true.

[00:03:16]So, what I now want my agent to speak back to me is not implementation details. I want it to speak product to me. I want us to have a two-way conversation, a proper conversation where we're discussing the actual product. Like, "Hey, we're building a social platform for dentists. So, what does this platform needs? How should it look like? How should it feel? What is the user stories? What should the dentists be able to see when they go here and here?" So, I want to get into conversations with AI using voice in such a way where it goes both ways. I share the product details. I get this conversation back. I get challenged. Hey, have you thought about this? Have you thought about that? And then eventually when I approve the plan, I actually don't care about the implementation details. And if you look at my boring stack, if you look at my TS Forge harness, my harness can generate full production scaffold, my boring stack in one go and it literally can't diverge from it because I had it running for 2 months, running eval's, building the same app until my harness had every single essentially self-improvement to guide DeepSeek-4-Flash to not be able to fail.

[00:04:36]So essentially, again in summary, I want to be able to talk to my agent just as I'm talking to my PM. We agree on the details and then agent starts building and it's constantly guided through my guardrails, etc., etc., etc. So now when you know what the problem is that I'm trying to solve, let's talk about technical details and I'm going to show you the demo or where I am right now. So if I go to as always to draw.io, essentially this is the use case that I want to do. I'm this person and I'm moving around the room. This is this brown thing is my office. I'm walking around the room all over my all over the space. I'm just walking. I'm walking. I'm I'm in front of my whiteboard. I'm you know, just in front of my desk. Maybe now I'm standing.

[00:05:23]Maybe I'm sitting. But I want to be able to speak out loud in this office no matter where I'm standing and I want to be able to have a conversation with my agents, with my apps, with my operating system, etc. without needing to be bound by the chair, by the terminal and I certainly don't want to be reading the walls of text. So I want to be saying, "Hey, today I'm building this app or I'm fixing these bugs. I'm going to be standing in front of my whiteboard. I'm going to be talking, thinking out loud, and my agent is going to be talking to me.

[00:05:55]How am I thinking to do solve this problem? Well, after some research of the omni microphone, so microphones that take uh the 360° audio, uh internet led me to this one, which is less than 100 euros. And then furthermore, I was looking for the case and I went to Printables, and then I found this uh awesome case that uh some awesome uh fellow has shared that both has a speaker and the placeholder for the microphone. So essentially, this thing you install, you get the audio both ways, audio in and audio out. Now, whether I'm going to use this case is unclear, but the idea is that I get this mic, I get a speaker with it, and then I can actually uh have it somewhere placed in my room, and I can just talk inside of the room.

[00:06:43]So, this is the this is the hardware that I'm going to get once I'm back in Copenhagen. This is how it's going to look like. And let me now talk about software and where the software is going to end. So, last night I was exploring uh some open-source solutions for text-to-speech and speech-to-text as part of the same API.

[00:07:02]I stumbled upon this thing called Speeches, which uh looked and sounded great. Whatsoever, after I did some basic due diligence, I came to a conclusion that this project is no longer maintained. And uh essentially, as you can see, these two threads are saying, "Hey, this hasn't been updated in months." So, I said to myself, "Okay, well, I I can't be uh depending I can't be building my uh my infrastructure uh on something that's no longer maintained."

[00:07:30]So, then I went ahead and took a couple of hours with the agents, and I built this voice voice box myself. In simple words, it's just a fast API. Why fast API? Why Python? You might say. That's because most of these speech to text and text to speech tool the libraries are written in Python. So, unfortunately, I had to use it. I'm not a fan of Python.

[00:07:50]I'm fan of Go. I'm fan of many other languages.

[00:07:54]Uh but short story, not to bore you with all the details, essentially, this gives us the voice both ways.

[00:08:00]Uh as you can see, it's using uh faster whisper.

[00:08:04]Uh etc. And then we have Kokoro and Piper.

[00:08:08]Kokoro, we go I would be using this on my GPU. Right now, I'm remote, so I'm not. So, essentially, right now I'm using this Piper thing that runs on my mini PC that I'm going to showcase in this very demo now. And the sound is a little bit robotic. That's because we're using Piper. Once I'm back in Copenhagen again, we're going to switch to this.

[00:08:29]But I'm going to demo it to you now. I'm going to show you how it works. And then you can get an idea of whether this is something that's good for you as well.

[00:08:35]So, first and foremost, what I'm going to do, I'm going to SSH into this Lenovo machine. So, if you remember from my previous video, if I go here, this is the guy that's running my Hermes uh Hermes agent and it's also part of my my K3S worker, it's part of my Kubernetes.

[00:08:50]So, on right now, I have SSH'd into this guy. I'm going to do B top just so we can see the resource utilization. And then you're going to see now once I start demoing you the uh the uh the the service, you're going to see the utilization of the CPU. So, I'm going to go to my open web UI and I'm going to show you how this works. So, I'm going to start the voice mode and then I'm going to show you how this works.

[00:09:13]Hi there. How you doing? I would like to learn Rust programming language. I'm a senior software engineer uh tech lead and an engineering manager today. I've been writing code for two decades almost. So, I would like to learn Rust maybe in a day or two cuz I can see that it's pretty uh well-established today.

[00:09:29]It's pretty popular. A lot of people are rewriting tools in Rust. So, I would like to get some info from you on what my pet should look like.

[00:09:45]>> Great question. With your background as a senior engineer and tech lead, you can absolutely get a solid feel for Rust in a day or two.

[00:09:53]Let me sketch out a plan. Your pet project should hit Rust's unique strengths, ownership, borrowing, lifetimes, zero-cost abstractions, and fearless concurrency.

[00:10:03]You want something that forces you into those concepts early.

[00:10:07]Here's my recommendation. Build a minimal CLI key-value store with these features.

[00:10:12]Storage on disk using a simple file format, concurrent reads and writes.

[00:10:16]A command-line interface. This will naturally teach you ownership and borrowing, passing data around lifetimes, referencing data in your store.

[00:10:25]The borrow checker fighting you and you learning to love it. Error handling with result and the question mark operator.

[00:10:32]Traits and generics making your store flexible, threads and channels for concurrency.

[00:10:37]Pattern matching and enums if you want to stretch a bit further, add a small TCP server so clients can connect remotely. Serde for serialization day one, syntax, ownership, the book's first 10 chapters.

[00:10:51]Build your key-value store. Day two, add concurrency, error handling polish, and the TCP server.

[00:10:57]>> All righty, so not to bore you as you can see, I showed you the CPU utilization.

[00:11:02]It used essentially as we can see here the voice box around the 1 gig of memory. This is essentially running in Python right now. So as you could see essentially, it does sound a little bit robotic, but it doesn't require a GPU.

[00:11:14]So as you can see, this is running on i7 6700T.

[00:11:20]So basically a CPU optimized for lack of power consumption. So a lower power consumption if you will. So as you can see, this is running on a 10 12 years old or more CPU. To me it sounds very good. Now, of course, I need to fine-tune maybe the speed at which the WAV file sounds. Maybe I should speed it up a bit, maybe 20-30%, so it's not so slow. But the point being is this is just the beginning of this journey where I want to fully do this not only in the chat. Right now, my agent is actually adding some client-side script for my harness, so I can utilize this in my harness as well.

[00:11:58]But moving forwards, I will be fully fully embracing the audio. Uh I'm going to be polishing the system prompts in my harness to to respond uh basically knowing that the agent will be uh talking, if you will.

[00:12:14]And I believe that cognitively I'll be building way better products because I will be focusing on all the edge cases, discussing it with the agent, all the details, all the decision-making, and then I'm going to let my tooling and my harness and my guardrails solve the technical problems. I think that's uh that's it for now. I think this kind of sums it up. I'm going to share this voice box in the comments below if you want to give it a shot. And until then, until uh the next video, I wish you all the best and uh have a great one. Alex.

[00:12:44]Bye-bye.