From Idea to Live Voice Agent in Minutes (Claude Code)

Added: 2026-05-06

[00:00:00]In this video, you'll get a backseat into how I built this voice agent with our new voice agent API powered by Cloud Code. It was a truly magical experience of putting in one prompt and getting a voice agent out on the other end that was ready in production. I'll show [snorts] you my tips and tricks so that you'll have everything you need to build great voice AI agents on top of AssemblyAI. Let's dive right in.

[00:00:24]So, this week at AssemblyAI is extra exciting because we're releasing our voice agent API. That's a big launch.

[00:00:31]What was the one moment during the build that made you realize it actually worked?

[00:00:35]>> I mean, for me was when I was able to go from zero to a fully working voice agent just by using our AssemblyAI MCP server and also just putting in one prompt into Cloud Code. So, the whole prototype was built in 20 minutes with a single prompt, just tweaking it afterward. What was the first bug that showed up and made you realize how fast [music] the loop was? I mean, for me it was um it was a bug related to Railway, which is where I'm hosting the app, >> [music] >> but I don't want this to show up in my LinkedIn post because it's kind of irrelevant to the voice agent API. You said it's about letting you focus on your expertise. What's the one thing you're most excited to build [music] now that the hard part is gone?

[00:01:18]>> Wow, I mean, I have a ton of ideas for voice agents that I'm sitting on, but the one that I'm most excited is actually this voice agent that I'm using right now. If you were to say this to a friend in one sentence, what would you tell them about why they should try this now? They should try this now because it's never been easier to build a voice agent on top of AssemblyAI's voice AI platform. So, the first thing I'd recommend is that you install our AssemblyAI MCP server. The full command to install our MCP server is in our docs and the link in the description. Once you have that installed, our MCP server will automatically pull our latest docs into your Cloud Code session. There's no more needing to search our documentation for updated information. All of that gets automatically injected into your Cloud Code >> [music] >> instance. All right, so kicking off our demo project, I tell Cloud Code to build a voice agent that interviews me and asks me questions and to help me write better LinkedIn posts. So, that's the big idea of this voice agent that I'm trying to build.

[00:02:20]Um as you can see, I copy and paste some docs links just to make sure that it [music] gets it just right.

[00:02:26]Um and I wanted to use the browser integration page as well.

[00:02:31]So, from there I just ask it to build a very simple demo that showcases the API [music] and I'm planning to host this voice agent on Railway and so I mentioned that from the very start.

[00:02:44]So, Cloud Code starts churning away at the code, building some front end and back end code as we can see here.

[00:02:51]>> [music] >> I've sped it up so that we don't wait too long.

[00:02:54]Um but as you can see in about 10 minutes or so, [music] we have a GitHub repository up and running with our voice agent demo. Um I go ahead and add it into Railway, uh which is where I've chosen to host my app.

[00:03:09]Now, I'm just looking up what command I need to start running [music] the app.

[00:03:13]So, I set the root directory to the front end as well as the back end.

[00:03:22]And I insert my AssemblyAI API key.

[00:03:26]So, I go on to our dashboard, sign in with Google. So, once I have my API key, I insert it into Railway and save and now we're ready to deploy our [music] app.

[00:03:42]From there, I need to add another instance, >> [music] >> um which will then host our front end and that's the URL that we'll connect to to see our [music] app and to play with it.

[00:03:53]So, this one has a root directory of front end.

[00:03:57]As you can tell, I've used Railway a couple of times and so I'm kind of familiar with the process, but what I've learned is that Cloud Code is actually really good at using the Railway CLI to do a lot of this for you.

[00:04:10]And the great thing about our voice agent API is that it's framework agnostic. It's a simple API that works with practically any coding language and any platform. [music] Um and so if you want to host it on Vercel, Railway, if you want to self-host it on your own server, um anything works because it's just so simple and so [music] easy to deploy.

[00:04:35]So, here now I'm setting the back end URL with the Railway service, um but eventually [music] I figure out that actually uh Cloud Code can just help do it for me.

[00:04:48]And after a little more wrangling and finagling of Railway, um in any second now I should be live with my voice [music] agent app. All right, so there's our app and as we hit start interview, we bump into yet another error of failed to fetch.

[00:05:07]So, this was because I think on the front end we set the [music] wrong back end URL to call for a token.

[00:05:14]Uh so, for context, um >> [music] >> if you're deploying the voice agent on the browser, you can make a request to your back end for [music] a token that you can use to authenticate your session on the front end.

[00:05:27]And that [music] way, we're able to start um the voice agent on the client side.

[00:05:32]Um without having to expose your API [music] key.

[00:05:36]All right, great. So, here we go. We have our very first voice agent in right just a little under 15 minutes. Um and yeah, it was really exciting to get it up and running so quickly, um but we're going to work on some improvements right away now that we see that it's working.

[00:05:54]Um the first [music] thing I'm going to want to do is to style it and kind of make it look like LinkedIn a little more.

[00:06:00]Um just to kind of nail that design and really get people into the mood of like writing something for their social media.

[00:06:09]So, I go ahead and tell Cloud Code it worked great. Um can you um you know, make some design improvements, styling changes, and we kind of want to make it look like [music] LinkedIn.

[00:06:24]So, we'll see fairly quickly um that uh Cloud Code is able to design um my voice agent to look like LinkedIn and lean into their brand and palette.

[00:06:36]Um and so after we push to get and refresh, we should slowly see um that yep, it looks like [music] LinkedIn. It's got those colors and the nice off-white background. Um and maybe we'll just start another interview to see uh how it works.

[00:06:58]Yep, it's working great.

[00:06:59]Um and so on to the next thing.

[00:07:02]So, one of our features of our voice agent API is that we support tool calling right out of the box.

[00:07:09]Um so, the tool that I choose to add to this voice agent is Exa Search, um which will basically allow the voice agent to kind of search up sources [music] to back up certain claims that we make in our post, um which will make it look a little more informed and backed by research. [music] So, I go ahead and get my Exa API key and insert it onto Railway while Cloud Code chips away at implementing Exa on uh the back end.

[00:07:42]So, I can see here it's uh made lots of progress.

[00:07:48]We should be getting a voice agent with Exa any time now.

[00:07:55]All right, so to test out tool calling, um I just started a new session and you know, stated a stat >> [music] >> uh just to see if the voice agent is smart enough to go and look it up using Exa.

[00:08:08]And sure enough, it finds this report by MIT on Medium, um which is great.

[00:08:17]Lastly, we're going to add some different voices to the voice agent. So, AssemblyAI supports a wide range of voices with more coming soon.

[00:08:28]Um and I want to let the user pick the voice that they choose uh for their session.

[00:08:35]So, what we'll do is we'll just add a drop down with uh all of the choices for voices with some sound defaults and >> [music] >> um in no time you should see that live as well.

[00:08:47]If you do end up trying our voice agent API, I'd love to know [music] what your favorite voice is.

[00:08:52]Um our team put a lot of work into picking great voices.

[00:08:55]>> [music] >> Uh so, let me know in the comments below if there's any voice that stood out to you. All right, so um Cloud has gone ahead and made the change and as you can see, [music] here's our full list of voices that are available.

[00:09:07]Um I just hit start interview with a new voice and uh there we go. There we have a fully fleshed out voice AI agent in, you know, very little time. Okay, the draft is on screen now. Want to tweak the angle or zoom in on a different moment? [music] >> No, this looks great. Thank you so much.

[00:09:26]>> You're welcome.

[00:09:28]Enjoy launching the API. So, on that note, it concludes our voice agent build-along today. I hope you enjoy what we we were able to put together and let me know if you do end up trying our voice agent API. I'm really excited to know what you think and if you have any questions, don't be afraid to reach out to our team in the comments below or on our website. Um our team is always happy to help. So, until the next one, I'll see you. Bye.

[00:09:57]>> Woo!

[00:09:58]>> [music]