AI Dev 26 x SF | Eda Zhou & Mahdi Ghodsi: Building Personal AI Agents with Open Source Models

本站添加: 2026-05-21

[00:00:07]So today we're here for a little bit of a different kind of talk. So this one is going to be more of a a hands-on workshop. So the first few slides going to be by my coworker here and uh then I'll take over the workshop part. That's a hands-on part. So my name is Maddie.

[00:00:26]So we're both from AMD and we prepared um basically more of how you guys can go from an open source model, deploy your own model on a GPU, connect that to your open claw and you know build something together. Uh so >> yeah, if you want to participate in the hands-on aspect of the workshop, make sure you're signed up for AMD developer program. We have GPUs available for you to follow along live during the workshop. And we also have custom lobster squishies uh for folks who finish a challenge.

[00:01:00]>> Hey, those are AMD branded lobsters, so you don't want to miss them. Um, and before we start, just a show of hands for me to know how many of you guys have ever used AMD GPUs for your AI workload.

[00:01:15]I see a few hands. Uh, a lot of potential though. So, today will be your chance. So we have limited number of GPUs that we're going to share the link with you guys later. Uh you can log in and access that notebook. So it is a chance for you guys to take out your laptop if you have it with you. We do have about 120 instances of dedicated GPUs for you guys that you can access.

[00:01:34]So while go through the slides, maybe it's good time for you to take off your laptop, connect to Wi-Fi, and be ready for the second part.

[00:01:41]>> I pass it to >> Thank you, Motti.

[00:01:45]>> All right. Uh, so if you want to follow either of us on LinkedIn, here are the QR codes. Um, this is my alter ego as a lobster.

[00:01:57]And so, real quick, uh, what is an LLM?

[00:02:00]So, we're just going to quickly go through the foundations here. So, an LLM is a model trained on a large amount of data. It predicts the next token and by itself, it's just a text generation system. So, it can answer questions. It can um generate and explain your code, but what it can't do is it can't act on external systems. It doesn't maintain its own persistent memory or manage a multi-step process. So on its own, an LLM just generates a response. It can't independently do these other tasks. And so this is the gap that we're going to focus on today and try and fill.

[00:02:37]>> Can I get the speaker notes?

[00:02:38]>> Louder.

[00:02:40]>> The speaker notes.

[00:02:43]Okay. And so as we step through it, we're going to step through filling this gap. So like we said, an LLM is trained to answer questions to predict the next token. A chatbot then has the conversation history injected into it.

[00:02:58]So it feels like it's remembering your conversation, but really that's just added into the prompt and it sees all of that in the input. Now the agent is where the next the real big step happens. So agent is where you start adding memory planning ability to call APIs uh take actions into loop until a task is done.

[00:03:21]And so really important part of agents is the loop. So the react loop stands for reason and action. And it's actually three parts. So reason, action, and observation.

[00:03:33]So with a loop um the agent can continuously think through a problem until it solves it. So if you think you ask a chatbot, hey, can you fix this problem for me? It'll give you a response, but you don't actually know whether that response is true or not, whether it works or not. With an agent, it'll actually call a tool. Maybe it'll call the code, call some test code, and it'll test if its fix actually works. If it doesn't, it'll go back and try something else. and it'll keep going through that loop until it verifies that the problem is solved.

[00:04:06]So with this loop, the base model uh is important, but actually a stronger loop could make your agent process even stronger.

[00:04:17]So there's three components to building an agent. Uh there's the model, the runtime, and the tools. And so breaking it down, model folks are probably familiar with here. This is the large language model. Uh there's examples of really popular recent models and all of these had day zero support on AMD Instinct GPUs.

[00:04:39]>> So the runtime is very important. It manages the loop context retries the state. Uh these are the guard rails.

[00:04:47]They uh know when to make the tool calls and then catch failures as well and to continue executing that loop.

[00:04:56]Finally are the tools themselves. These are those external capabilities. So, MCP servers, uh, APIs, CLIs, and these are actually usually abstracted from the model itself so that you can use a lot of different kinds of tools. It doesn't need to know those details. And MCP servers, there's plenty of open servers.

[00:05:16]Uh, if you've been to Mott's previous workshops, you'll know, you're familiar with the Airbnb one. Uh, there's GitHub ones. Uh, there's plenty of ones that you can just use right out of the box.

[00:05:29]And so finally, how is Open Claw different than just these agent frameworks? So a lot of agent frameworks are just libraries. You import them, write loops. Uh they're reactive, so they wait for you, and they only run when you run the script. Open Claw is persistent and is an application. And so it's persistently running on your machine. And so if you want a cron job or a heartbeat monitor, it's just an input to that application. Whereas if it's a framework approach, you need to build a scheduler around it. You need to build more uh stuff around it in order for a cron job to actually work. And so the open claw approach, it just works because that's how the architecture is structured.

[00:06:12]And so without further ado, uh if you want to participate in the hands-on portion, uh you need to go to this URL.

[00:06:20]Take out your laptops, go to the URL, and it will start spinning up your workshop. We'll give everyone some time to get this set up, and then Marty will go through it live.

[00:06:31]>> Yeah, we'll leave this up for a couple minutes. So, please, if you have your laptop, log in. As I mentioned, we have about 125 dedicated MI uh 325 available for you guys that you can actually go deploy the model, then connect that model to your open claw agent that we built together today. Uh it is kind of a first first come first serve. So if you have your laptop, I believe not all of you guys have laptops. So a good portion of you guys will be able to access this today. Uh so we leave it up a couple minutes and then I'll switch to the same notebook that you guys going to be accessing today. So if you guys um have never done any uh model deployment on GPU, that's actually another question for me just to know how many of you guys have ever deployed a model uh on your own GPU or hardware before.

[00:07:17]Okay, that's a lot more. That's great.

[00:07:19]Um, so to do so, you usually need some sort of framework or a serving framework to do so. So today, we're going to use VLM.

[00:07:29]So VLM is a very uh popular framework right now made for um GPUs without you know restriction to any specific brand.

[00:07:38]So uh we're using VLM as one of the most popular serving frameworks. So to do so you will have access to you know this notebook that has all the instructions available for you guys. If I want to start this uh model so right now we're using 3.520B I can just copy this which is going here on the terminal and paste it here.

[00:08:03]So what is this doing right now is as simple as VLM serve model name. The model name comes directly from hugging face. So, if I just go here on hoging face and look up models and put Quinn 3 uh.520B uh this one right here. So, you can see it's actually the exact same same name that I got from here. I could just copy and paste this uh right as the model name. And that's all you need basically just BLM serve model name. And this works on AMD GPU without needing to do anything. It's just BLM as a first class citizen on AMD hardware. So you can use that. The rest of them are optional parameters I'm showing here to enable tool calling which is required for agents to be able to actually call the tools. You do need to have the extra parameters such as a tool call parser which comes directly again from the model provider. So it's a model based and API key is for the software such as open cloud to you know add that level of uh security. they usually need you to add the API key and that's for that purpose. The port number again is in for us to have as an optional added all those and that's it. So once you run that it will start to go and load the model weights on your GPU. Here we have a AMD GPU uh and it will basically uh load the model the model weights uh go through all the VLM serving uh requirements that we have and once it's ready you're going to get you know the three lines of hey the model is ready for you while this is happening I will go ahead and run this uh next cell this next cell is all it's doing is just checking to see if the server is ready so it's nothing important I don't want to spend time on this but we will know if the if the server is going to be Meanwhile, why this is happening, we'll get to open claw parts. So, Open Claw um basically if you guys never used it, it's a very personable agent. You install it on your laptop or your machine and it will have access to all your files. We're not going to talk about security part of it. you know that's that's for them to figure out but it does have a lot of power and access and um you know you can disable any sort of access you want but it starts with a lot of power uh such as your files bash bash commands you can give it access to your emails or whatever you want but you can always you know go back and say I don't want you to access that so to to you to start openclaw they made a really nice uh openclaw onboard u command for you guys that you start with so I was hoping you guys can do it. I I see maybe a few of you guys may be doing this, but uh feel free to stop by and we can show you how to do it on your own uh time.

[00:10:45]So, on OpenCloud onboard basically uh walks you through all the parameters you need to set and it's an interactive u session. It will take a couple seconds for you guys to load. Uh but meanwhile, we can also check to see if the model is ready. It's already loaded but and the model is ready. So this is the server check I had and this is basically the VLM server command we ran. So that basically means the model is ready for you guys to use.

[00:11:14]Going back to uh what I had here for the open cloud onboard first is basically accepting the risks as I mentioned we're not going to talk about that today.

[00:11:26]And I'm gonna reorganize this so you guys can see the side by side.

[00:11:34]Okay. So except yes. Uh that will walk you through the next part. So we have provided all of this for you guys. So when you're on your own, you actually you don't need me to tell you all of this. It's all available for you guys to go through. But we'll do it together here. Uh so the model provider the really cool thing about open-source ecosystem such as openclaw vlm sg lang is they're working well together and now uh with openclaw you don't need to actually go do any sort of you know weird configuration because it has it available for you guys. So if you go down all the way to here you should see vlm and uh other frameworks available such as sg lang which is another very popular uh serving framework. So, VM is the one we chose. And you can see uh the port we chose 8,000. So, we're going to stick to that. The API key for simplicity put APC 123 and the model name is 3.52 B.

[00:12:36]So once you configure the model, that's basically all you need. If you look at the next couple of questions I have is all a skip and that will get you started immediately to be able to use openclaw with all of its power. But one thing to note is uh open claw basically is as powerful as you make it. So the tools that you give it access to is what makes open claw do things for you. Um without that is limited. It still works. We're going to see some of the examples but uh that's for you guys to on your own time.

[00:13:06]Again, if you want to set this up, spend some time, see what kind of tools you want to give it access to. So, basically, I skipped all the questions of what channel, you could do, Telegram, you could do WhatsApp, you could do all your all your messaging apps. In terms of uh provider, uh we're going to skip for now for uh any sort of search providers they have. the skills I mentioned, we want to skip that for now, but you can actually enable a lot of skills such as like do my Gmail access or um there's many other skills that you can look up online uh that might be useful for you guys, but uh you know Gmail access, calendar access, uh web browsing, a lot of those are available for you guys to choose and hash and terminal will make sure that our open claw is up and running.

[00:13:48]Okay, the first time you run open claw, this is kind of where it gets you. Um, it shows how personable it is. So, the way that works is the creator of open claw created all these MD files for you guys. Soul.md is the soul of your agent.

[00:14:04]Basically, you tell it how to behave and it's really important and actually depending on the model. I've had very interesting experiences with soul.md.

[00:14:11]There was a time that I I wanted to make my agent to skip bunch of verifications but in my solar MD I put do not believe anything do your own verification. So no matter what I did it actually did not accept anything I wanted to do. It will go back to solar MD that overrides any sort of rules that I give it. So this is one of the most important files in your open cloth.

[00:14:32]Agents MD is basically all the rules of how you should behave, what you should do and that comes uh packaged with open cloud. The identity MD is basically uh who um you should be you kind of emojis that you want. We're going to go through it right now and you're going to see how that's going to be created. So while I leave this up uh I'll start interacting with it and then we'll show again these files.

[00:14:57]The first thing you should uh basically do is introduction of yourself and introduction of the agent to you. So here for example it says hey I just came online who am I who are you? So I'll call it your rock roll and if you don't know rockim is AMD's equivalent to CUDA.

[00:15:15]So you're rock and roll. Uh my name is Maddie.

[00:15:21]Your vibe is uh rockstar.

[00:15:27]Uh I'm in Pacific time zone.

[00:15:33]Let's start.

[00:15:40]So what is going to do right now? It will start overriding all these files I mentioned. So the soul MD, uh the identity of MD, the user MD which is like basically uh who I am uh any sort of tools that I wanted to give it. I basically talk to it. I don't need to do any sort of configuration. That's one of the really most powerful things about OpenClaw is even if it has a problem, you don't need to go figure out what the problem is. You tell it. You say, "Hey, you have a problem. I want you to basically fix this problem." And it actually go and configures itself. It knows its own source code.

[00:16:11]So again, it's kind of gone back. It wants to be very personable with me.

[00:16:14]It's like what matters to you the most?

[00:16:16]You know, any hard boundaries? anything that you want me to, you know, take care of it for you. So, I'll say, you know, uh just be helpful.

[00:16:27]>> Helpful and don't ask many questions.

[00:16:36]>> All right.

[00:16:38]So once we answer these question that bootstrap.mmd which is basically initial md file for it will be deleted and it'll be set up for you based on what you actually ask it to do. Um and this is where I wanted to print some of these files that you see. So it says okay I deleted my bootstrap and you know I'm ready to roll for you and it has a vibe of you know that rockstar that I asked for it and you can see all of those here.

[00:17:03]So basically based on what I answered it's no magic. It'll just keep notes of your stuff. The software is really smartly designed to be able to read this and always be available to you guys. So the identity of MD is basically what you know it should be called. Solid MD is what I want it to be like the vibe for it. The boundaries that I wanted to get the vibe again uh what else? Ages MD is something that I didn't change. We'll change that in a bit to show you how that works. But basically that means it's all set up for you guys. So that's for the initial setup. Now let's let's show some examples what we could do with this guy.

[00:17:42]So agent MD as I mentioned is the rules for the agent. I can ask it to be more of a you know a debugger for me for my code. So if I have a buggy code I want some sort of policy just don't invent anything. You can type all of that in MD or I can ask it. But I want to show for completion. You can do both. You can go mess around with agents. MD or you just basically ask it to do. This one is the one that you actually go mess around with agent MD. So if I run this, it'll actually overwrite this in my agentmd with some information that I want.

[00:18:10]Uh so now it's ready for me to actually be used for debugging. So for fun, what I had planned for you guys is um a little application that I made with a obvious bug in it. So this is basically an application on my GitHub. It's not on this system and I want to show how easy it is for you guys to basically drop a GitHub people. If you have no coding experience, if you don't know how to set up stuff, you don't need to. You could basically just like drop it here and say, "Hey, uh, go figure out this project, install it for me, and run it or like make it ready for me." So that's all I'm doing. And it will basically go look at the GitHub repo, pull it on your computer, installs it for you, and tells you how to run it. U, okay, so we got this installed for us and actually tells us how to run it. It says just go to this this folder here. It's ready for you guys to use and I will do that. So just to show you guys oops it's hard without a mouse but we got it.

[00:19:08]So basically if I go here and open terminal install it the instructions told me that all you have to do is just do something like this python main again copy pasting and let's do a simple 110.

[00:19:26]So this is a word per minute counter. Um I made this like basically vibe coded all of this to see if you I can make an application and uh I'm just going to like type random stuff here.

[00:19:38]The problem is >> does anyone see >> there's a bug right? Um and I intentionally introduced a bug to show you guys how you can basically get this to debug for you and fix the code.

[00:19:50]There's a really obvious bug here. I got a zero accuracy. That was the intentional bug I had for it. But just to show you guys how EDG is is for openclaw which is you know it's not sold as a coding agent as a general agent can actually do that depending on the model you use and another thing I've heard is like well open source models are not as good I mean depends how you define they're good enough they're good for a lot of use cases such as the one that I'm showing you right now. So if I just like b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b basically say, oh there seems to be a a bug with this uh open type tester I think it's called it. Um can you check?

[00:20:29]So basically we'll be able to look at it and try to like debug the the problem.

[00:20:36]And this is all again on an open source.

[00:20:39]Uh, okay. I should copy paste that. It's looking as a different bug for >> uh Yeah. No.

[00:20:50]>> Okay, let's do it.

[00:20:55]So basically it should be able to go figure out the the test files for this and see if there's you know anything that is getting wrong and read the code and try to fix it which you can also if you find anything useful for you that you need to instruct your agent to do over and over this is a very very uh simple use case that I'm showing which is kind of struggling right now for me run the test and see the problem in in my application that you installed.

[00:21:35]All right. So while this is figuring out so um basically what you could do with uh openclaw hopefully this gets it but what you could do with openclaw is you say you have a task that is very redundant right like for example go on this website and go read this material and extract this and then go you know email this to this person and do that. So you have a checklist of stuff you need to do um you can actually ask it to create a skill for you and that skill becomes available every time that you want to use it. So it actually uses the skills for it for it that would save that and skills with MD. So the agent every time you ask a similar thing for it, it's like well I know what to do. I'll do it for you. And that's basically what we wanted to show you guys here that uh with this um sort of problems like this one that let's see if it found it.

[00:22:30]Okay, so basically he's found some sort of problem with it. It says it was showing 0% so 80% accuracy. So I know the problem uh that now apps work correctly. So I can fix it for you guys.

[00:22:40]So basically it did the it did what we wanted. It fixed the problem and now we can say okay now go create a skill for this. I have some instructions for it that the skill I want to make sure that you you build is doing one two three and you know the instructions are up to you and you're trying to show how you can get it to create a skill and then reuse it later.

[00:22:59]So I was just prompted it and given the instructions you want. I can create the skill and um we'll test it in a second to to show you guys how it's working. Um for completion again all of these are within the environment of open claw. So once you install your open claw such as uh the one we did everything resides in this openclaw folder and you can always go check it. If you don't like some sort of policy you made you can always go manually delete it. So now I created skill. We can check to see if it's available.

[00:23:31]Here we go. I ran this and this is what it's doing. Showing okay under skills pyest debugger is the name of the skill I created. This is what I will do and you can use it anytime in future.

[00:23:43]So for using the skill uh openclaw has a lot of ways to do it. You can you don't have to like remember exactly what you thought. You can literally naturally talk to it. But if you want to enforce it, you want to make sure that it's definitely doing this. It has a lot of shortcuts too. So the slash that you put will give you a lot of options that you can see is it's showing me right now like help or gateway status. But if you type skill, it says okay, type the name.

[00:24:09]So I can just type the name of the skill and give it a new uh problem to solve and it should remember what it you know it learned and apply that to the same the new project that has never seen the code for. So what I've done here says skill use the name that I just used pi test debugger to create a skill. Go create go just like look at this GitHub repo. I didn't even tell it what this is. I didn't even tell like go clone it.

[00:24:34]It's going to automatically go through the loop that it showed to reason. It's like what is this guy doing? He's giving me a GitHub repo with no instructions but I can basically go figure out what it is based on the skill I have. So while this is again applying the skills uh that we we talked about to figure out this this new problem um one other really cool thing with u openclaw is how it could be a multi- aent system and do a lot of stuff with you well the agents can communicate with each other in an environment recently I was working on a different project this is for you know our internal use that hey uh whenever we run any models on a GPU you can actually increase how fast that model can run by going down to the very low-level codes but it's a very complicated problem. So you know it has a lot of steps and even for us it's one of the things that we have challenges all the time across the hardware companies. So while we doing this that you know I'm basically kind of showing you guys a little visualization to make it fun for me. I had a little you know uh pixel office that each of these agents basically you see are my agents that do some sort of work. One of them is my benchmarker. One of them is my analyzer. One of them is my uh skeptic.

[00:25:50]That is really like hard rule that do not accept what they pass to you. Go test it yourself and make sure it's good. Um you have your CAM bond that you can always observe. So you can get a little bit fancy with this. Once you have it ready for you this is just like you know showcasing that for myself. you know, I ask it to go run the optimization loop for this specific model for my AMD Mi325 GPU or 350 GPU and it basically goes through the loop of hey, I'm going to call the benchmarker first. Then I'm going to go uh you know get the benchmark numbers and pass my analyzer. Analyzer found that you know this is the good kernel to actually go and optimize and then later on the analyzer passes that to another one that is optimizer. the optimizer is supposed to go write the code and make it optimized for you guys. And finally, there's a skeptic that will be able to basically um uh analyze the final results and tell you if it's a good or good result or not and passes back to me. So, this is this basically ran over I think maybe an hour or so. I just left it. I came back. I looked at the result.

[00:26:52]It looked good. I showed it to my boss, got a pat on the back and moved on. So, this is like an example of what you could do.

[00:27:00]Going back to my uh to my agent, you could see I applied the fix. Uh now I found basically the the exact problems I had. It fixed it and came back to me and show me the results. So speaking of the multi- aent part, I'll show you guys an example of that as well. That'll be a wrap for the workshop part. I did have a challenge. I don't know if you guys are running this with me. I will uh allow you to fix to to see if you can, you know, finish the challenge. But uh this is the the multi- aent part of it that I want to make sure cover. So to add a new agent um the one like the one I showed you guys have multiple agents. It's basically as simple as open claw agents in the model and the agent name. So one of the most uh one of the most popular way of using openclaw is creating morning brief. Say you know anybody has their own unique interests and you want to make sure that it's customized. Every morning you wake up to something that you like whether it's your financial news, whether it's your GitHub repos that you're watching. This is right now be u I'm I've been told this is one of the number one use cases of open claw.

[00:28:07]So we're going to make that together right now. Um so the way you can actually add multi- aent is by basically doing openclaw agents add and agent name. It will walk you through a very short uh onboarding again. Do you want the workspace to be here? Yes. Do you want to copy the off? Yes. Uh configure model off for this agent now. No.

[00:28:33]And it should be able to basically start the the agent bootstrap again for me to know who it is and what we want to do.

[00:28:43]Uh configure channels. No. All right. So the agent is ready for me to use. And the way I can use that agent within the same agent I had now I have a main agent and I just added a second agent is again if I just do slash agent and I put the picker it's something like I don't know if you guys see it it has a little like name here. So main is my default now the morning brief. So I switch to my morning brief and it says uh like hi you there.

[00:29:15]So it should basically start again. It's like a morning brief and now I give it a task that um you want you want to basically be the guy that always work with me to have the best uh morning beneath that I have because over time you can actually go prompted to say like I don't like this or I want to make this modification. So that's why you're actually creating a a second agent. Um, do you guys maybe again if you guys know why I would not just have one agent to do everything for me and just I need to actually have multiple agents because you know it's connected to the same model. I'm not making a new model. The answer was right. So it loses context.

[00:29:51]Yes, because that's the whole nature of LLMs and he was right too. So there's one down there. I guess uh the whole nature of LLM is attention based. So based on that you have a limited attention to the to the model and the context length of it. So even if you have a lot of instructions for one agent, you're not going to expect that to have the most powerful response for you all the time because you know you have given too many things. That's why you want to separate them in in its own environment to make sure they have access to their own tools and their own purpose. That's one of the reasons that you want to maybe have a morning agent dedicated for that task.

[00:30:26]So again, uh my name is Maddie. I'll just like basically do a very quick thing here. Uh you are morning brief agents. Um don't ask more questions and get ready.

[00:30:47]So as simple as that this could start again working for you and it will be able to uh create that morning brief that you want for your you know whatever use cases here what I have for example is for my own use case I want to know any changes to SG lang project which is a GitHub project I'm not telling it where it lives it needs to go figure out how to get that information BLM project again go figure out where it is find out the information for me hugging face transformer is and another project that I'm interested and at the end I say like you know also like a summary of latest AI hardware news so go again read the news yourself tell me what it is and it will create like a version for me and um based on that I can like prompt it again and say hey um this isn't what I like this is what I'm not don't like and I will have again configurations to be able to maintain that and and improve it over time so this is going to go through that and do it uh probably respond to me in a minute. So, this is doing all of its research, doing all the reasoning. There is ways that it can actually prompt the open uh claw agent to do um the verbose mode where it actually prints out a lot more information for you that like what I'm doing, what I'm looking at. Uh the simple way that when you set it up, it basically goes the most basic uh implementation of it, but you can again, as I said, you can um prompted to figure out what the problems with its own configuration is. That's a big hint for the challenge part because the model we have here coin 3.5 is a multimodel. It can do images. It can do text and sometimes when you start the agent it only configures as text right. So for me to start I was basically go say hey you have image abilities go figure out your configuration and try to fix your image reading uh problems.

[00:32:43]Well this is still working. There's more um commands that I can also tell you.

[00:32:48]One of the most annoying thing is when you, you know, have something that is on a wrong track, you want to stop it.

[00:32:54]There's keywords such as stop or abort that you basically can ask it to do. So, for example, a stop if I type it. Um maybe I'll just do it for for the purpose of this while it's doing its uh thing just to show you guys. Or maybe should I wait? I don't know. I'll show you guys how to stop it. So if I do stop or say abort, it will basically take that information tries to interrupt the current process and um stop the running process at that moment. It is not very good at that. I'll give you that in a bracket. So sometimes you have to do it like multiple times to do it. That's one of the weaknesses of it and there's known issue for it. But it's available that works sometimes.