AMD Ryzen AI Halo - 100% Local AI

[00:00:00]Okay, so a few months back I made a video about running local AI on an AMD workstation with the Radeon AI Pro R9700.

[00:00:10]Now that's a 32 GB card and it pretty much handled everything that I threw at it, but it clearly had its limit. It was 32 GB. You really couldn't get any models bigger than that on the actual card. And when that comes to something like a 100red billion parameter model, it really doesn't matter how clever you get with quantization, you're just not going to be able to get something like that to fit. So today I've got something a bit different. This is the new AMD Ryzen AI Halo. And the big thing here is that this has 128 GB of unified memory.

[00:00:45]So that means models like GPT OSS 120, some of the bigger Quen 3 models, Quen 3.6 six models, etc. that I simply just couldn't load last time or I couldn't load decent resolutions of them. They can actually run locally on this box.

[00:01:02]Now, full disclosure, AMD has sponsored this video, but as always, everything you're going to see, all the demos that I'm going to run are me running them on here, and I'll show you honestly what you can actually do with this machine.

[00:01:14]All right, so before I show you demos, it's worth understanding what's actually different about this machine because it's not just a bigger version of the last one. It's fundamentally a different architecture. With the previous workstation I showed, you had a discrete GPU. So, it had its own 32 GB in that case and the VRAM was super fast. But once you went over that with a model, you basically were left to offload layers to system RAM. And even though that machine had really fast RAM and a good amount of it, your token speeds would basically fall off a cliff once you weren't using it on the actual GPU.

[00:01:48]Now, what the Ryzen AI Halo does instead is it uses unified memory. So, the CPU and GPU share one big pool of memory, and it's got 128 gigs of LPDDR5X in here. And the cool thing is it's up to you to allocate how much you want to give to the CPU to be able to use and how much you want to give to the GPU to be able to use. And because a lot of the applications we're going to be running don't require that much system RAM, you're going to see that you can allocate the vast majority of that RAM to the GPU where you can use it to load models. All right, so the chip driving all of this is the Ryzen AI Max Plus 395. It's got 16 CPU cores. It's got a Radian 8060S GPU that's about 60 teraflops at FP16, but it's also got an NPU that comes in around 50 tops. Now, currently, most of the LLM stacks and a lot of these apps are actually not making use of that MPU, but watch this space. I do think you're going to see this start to be used a lot more in the future. Now, interestingly, AMD has already announced that there's going to be a Pro495 version of this coming with support of up to 192 GB of memory. So, that one could be really interesting to look at in the future as well. Now, as I'm going to show you, there are some trade-offs with the unified memory, but for most of the applications I've tried, it does allow you to just load much bigger models or to use multiple models at the same time. Okay, so let's talk about the out of the box experience.

[00:03:20]There are actually two versions of this machine. There's one that is a Windows version, and then there's the one that I've got here, which is the Linuxbased version. So the cool thing here is that unlike the previous workstation, I didn't need to partition a Linux drive.

[00:03:35]I didn't need to install Rockom and all the drivers and stuff like that. This just comes out of the box set up for this. In fact, it's got this piece of software which is the AMD Ryzen AI developer center. The idea here is that this is where you can come and see like all your system software. You can see what versions of PyTorch you've got, what versions of Rockom that you've got in there. You've got a bunch of different AI apps out of the box. So, if you want to run something like Llama CPP, you want to run something like Comfy UI, you can actually do that out of the box here. You've also got all your settings in here that you can come in and play around with. Now, the big one is going to be the graphics performance, right? So, this is where you can basically decide how much dedicated VRAMm do you want to allocate and how much do you want to basically decide that for itself. And you can see here that the current selection we've got 75% of this set to actually to be shared by the GPU. So at any point if you want to change that, you can come in and change that. You've also got the ability to set this up to use it via remote access. So basically using it by SSH and that's one of the ways that I've been using it while I've been testing it. You don't have to set this up just to be sort of your main computer. You can actually go and put this on your desk somewhere, have another computer SSH into this and have it just purely running AI models there in the background. All right, the other stuff that's in here is a lot of the resources. So, they've got a whole user guide of basically how to set the machine up, how to configure the BIOS, how to do a bunch of things like that.

[00:05:09]They give you access to the AI developer program in here. But probably the big thing for a lot of people about this developer center are the actual playbooks. So if you come in here and click on playbooks, you can come in and see that they've actually set up a website of how to do different kind of tasks. Now you can pick whether you're on the Windows machine, the Linux machine. Obviously here I'm on the Linux machine and you can see if we start up with just a beginner's one, a lot of them are kind of obvious. you want to get Comfy UI going and use it with one of the models that they've already got pre-installed here, you can run this and because this is built to be a general AI machine out of the box, things like Comfy UI just work. You don't have to worry about getting the Rockom version.

[00:05:53]You don't have to worry about which version to install. And I'll show you that while this comes pre-installed, you can certainly still install the models that you want and use them via code if you want to rather than the UI. Now, other things that they've got in here are obvious things like running and serving LLMs with LM Studio and even things like setting up PyTorch with Rockom so that you can use that. It certainly gets a little bit more interesting when you look at the intermediate notebooks in here. And you've got things like fine-tuning with Unsloth. So, many of you have heard me talk about that for most people, Unsloth is probably the best way to get started with fine-tuning, to be able to do basic fine tunes. And here they've got a playbook of walking you through it. And not only that, it's got information about how to configure the graphics performance for different kinds of tasks in here. And then it gets really interesting when you start to look at some of these advanced ones. So there aren't a lot of these at the moment.

[00:06:45]Hopefully over time they will create more of these. So while there are ones here for clustering to AI halos, big thing in here is if you wanted to learn about writing your own custom GPU kernels, you can do that in here.

[00:06:57]They've got a full tutorial of how to do it, of what it's actually doing, but also how to set this up on the machine, how to configure it, that kind of thing.

[00:07:06]So, while no machine is going to be perfectly configured to every single thing that every single person wants to do, this comes out of the box with a lot of things that a lot of people are going to want to do with this kind of machine.

[00:07:18]All right, so let's jump into some demos and just see what we can actually do with this. Okay, so clearly large language models are where probably most people are going to be interested in using a machine like this. Now, so I've installed LM Studio here. When you basically install this, you want to make sure that you come down to runtime and actually install the Rockom Llama CPP version in there. That doesn't come in here by default. And of course, if you want to actually use this with code where you're actually pinging this, make sure you've got developer mode turned on as well. All right. So, the first model I thought we'd start off with is a 9B model. So, this is the Ornith 9B model with a MTP drafter in there as well. And I covered the big brother of this on the channel before. And it had very nice sort of thinking when we looked at it.

[00:08:07]And the outputs were pretty decent as well. So, just looking at this, I'm really trying to just get a sense of how fast does this model actually work in here rather than the quality of the model itself. This should be the speed that you're seeing for most of the 9 billion parameter models where you've got a multi-token predictor drafter working with it. And nowadays the cool thing is that models like Gemma 4, models like the Quen 3.5 3.6 series people are actually releasing them with drafters as well. So you can see it's going along pretty nicely there and we're getting just below 40 tokens a second on this dense model. If we now move to a 35BE with the 3B active. So this is the Quen 3.6 with a drafter, you're going to see that with this we can actually even get better speeds because we're not actually using a dense model out. We're using a mixture of experts model. So this is definitely doing really nice fast token prediction here. And while it may not be as fast as the AMD machine I showed you last time with the dedicated GPU, you're still getting easily above 50 tokens here. And so this particular model is one of the models that people are actually using for local coding and for tasks like that. I should point out that the quant on that one was actually an Excel quant as well. All right, if we now go to the big GPT OSS 12B, let's see how we actually do on this one in here.

[00:09:34]So this is one of the models that comes pre-installed on the machine. And remember that this is also a mixture of experts model in here, right? So we've got it doing the thinking actually much more succinct thinking than we saw with the Quen models etc. And then we've got a very sort of open AI style of where you've got very nice markdown. You're getting all that showing here. And we could be exporting all of that out if we wanted to save it, that kind of thing.

[00:09:59]And you can see for this model we're around 30 tokens a second. Now, for general use, for chatting, for doing rag applications, this is a decent speed for what you want. For things where we're using it for agents and stuff like that, we probably want to go for something that's smaller, that's something that's a bit faster, etc. So, let me have a look at that use case in the Hermes agent. Clearly though, this is having no problems running models that the previous system just didn't have the memory to be able to run in the first place. Let's jump in and take a look at doing some image generation and video generation. Okay. So, one of the uses that I've gotten really interested in with using this machine and recently is the whole use of sort of making artificial videos. And while there are lots of good proprietary models out there that can do this from the likes of Google, etc., they are not cheap, right?

[00:10:51]They get very expensive and especially if you want to do lots of different runs. So, one of the things I've been really interested in is how could I make something where I'm building my own sort of little movie system of where I'd be able to code prompts going in, then mass- prodduce a lot of images, pick the ones that I like, and then use them as source material for particular video clips. And while this is still an idea that I've been playing around with, what I thought I would do is just try this out with this machine. And obviously, the big candidate here is Comfy UI. So, cover UI is installed by default in here. If we come in and actually just make sure we've got the dev mode turned on in there, that will actually allow us to be able to access this via com UI's API so that then I can use code to actually do this. So, let me show you sort of what I've been doing. Right? So, it comes pre-installed with the image Z model, which is a pretty nice model, but you can see that like that can make a nice sort of painting, can also do pictures and stuff like that. But also, you've got the ability to download different models, whether that's video models, so things like like LTX or the W models, which I think I showed in the last video, which I'm still kind of surprised actually how good these open-source models are. And if you then start fine-tuning them, you can get them to be actually even a lot better for specific styles that you're interested in. What I wanted to do was I wanted to use the Quen image model. So you can see in Comy UI, this is basically all set up. All I need to do is download and put them in the right folder. And then I can come in and start to render out my images. Okay. In this case, what I wanted to try and do is on Instagram, you see like all these pictures of cute little kittens dancing. So I've got an idea how they do it. I think they're using pose estimation, copying the poses from humans, and then using a control net to be able to put that across. And that's something that you can actually do in here. If you come in here and you use a cany edge detection control net and I'm sure if you go looking around you'll be able to find a pose detection control net as well. But what I thought I would do is at the start just give it a prompt. So I took a screen grab from one of those kind of videos. Got an LM to analyze it and give me a prompt for a basic prompt for it. And then I got it to make a bunch of variations on the prompt in here. So you can see here we've got our little kitten that we're able to render out. And then I coded this up of where it would basically generate lots of different output pictures. And I started doing it with one of the default prompts that they had in the Quen image sort of settings in here. And so this was the basically the same prompt just slightly changing the seed. So you can see just each time I've changed the seed. That was the kind of idea that I went with. But I actually coded up a little script that would actually change the prompt a little bit to change the pose. So in the end, we ended up with like a bunch of these kind of little kitten poses that we could then start to sort of render out. So you can see we've got disco point, we've got spin, we've got hip-hop. All of these were decided by the language model to actually come up with different images etc. Now rendering on this model with these images was taking I think about 19 seconds 19 20 seconds per image in there. So originally I titled the folder that I put in this in overnight image/v video creator because I thought it was going to take a long time. Actually it turned out it didn't take that much time at all to be able to render some of these out. Then what I did though was take these and put these into the LTX video model. And again, because I was doing this with a script, it's all automated. So, literally, I could just leave it and it would go for a few minutes trying out to generate these dancing kitten ideas. And so, sure enough, leaving it for not too long, pretty quickly, I had a bunch of these, which I could then start to see, okay, how was this going to work as a project?

[00:14:49]So, if you look at it, you can see that, okay, my dancing kitten's maybe a little bit slow, but it's on two feet. It's doing what we want. And so from this I could decide whether I want to change the prompt or whether I want to manipulate other things. And one of the things that I wanted to try was doing much longer generation. So rather than just generating sort of 5 seconds, if I go out to sort of 12 seconds, how does that actually go? And you can see that, okay, we're getting on point that we can certainly render a nice background. We can render the kitten. We can render some of the dancing going on there. Now, the thing for me that's cool about this is that this was all done programmatically, right? This wasn't me manually coming up with prompts and trying to do it. So, the idea for doing this kind of thing is not that I'm going to make one or two of these, but I'm going to run it overnight and make like 500 of these and then be able to go through and pick out the ones that work best. We all know for things with generative AI nowadays, a lot of it is about curation as much as it is about generation in here. So for me, this is a really nice use of the Ryzen AI Halo is that when you've got something like this, I don't need to worry that it's costing me 10 cents per second of generation and this was a$120 to generate or something like that. That's just not the case here, right? I can run this overnight, generate hundreds of these, and then come back and curate the ones that I want out. And that's both for still images and for videos. So, I don't know if I'm going to continue with doing the dancing kittens, but I'm definitely interested in the idea of making short form videos of 5 to 10 minutes that tell a story where the models have actually blocked out what the scenes are. They've generated the first images for the scenes. they've maybe generated images for the surroundings, etc., and then being able to render a lot of them out over time, and put the final thing together to get a nice product out of this. So, for me personally, this is a really big win with this kind of machine that you can just leave it on, apart from the power costs of running it. You don't have to worry about paying API costs or token costs for generating this kind of thing out. Okay. So if you want to run something like Hermes agent or open claw, you can certainly do that with this machine. So here you can see I've basically just put in a quick install of Hermes agent. I'm running this through LM Studio. In this case, I'm using the Quen 3.635B model with speculative decoding on there for the MTP. You [snorts] can see that this can go along and can actually handle function calling, can handle running the skills in there, can handle a whole bunch of things like that. Now, you can use this like I am here through the command line, but I'll show you in a second, too. Of course, you could just spin this up as a web UI if you didn't want to use it via the command line. So, let's look at how to do that. And you see now I'm using it from here. I can then basically change models if I want to. pretty easily. I can just come in here. If I wanted to use something like open router, I can just configure it so I can use all the open router models in there. But if I wanted to stick with the LM Studio models, you can see here I've got a number of different models. I've got this ornith model in here. And so this is a model that I covered recently on the channel. Turns out this model was actually quite okay for Hermes stuff as well. And sometimes you'll have it where it takes a little bit of time to load the model. Turns out actually at the moment I've got both the 3.6 Quen loaded and the Ornith loaded in there. So it's allowing me to switch back and forth between those. Now don't forget with Hermes you can actually have it where you have one particular model for a particular skill or sub agent. So that's something where you would be able to have a specialist model if you wanted to have it for coding. And the cool thing is because we've got quite a bit of VRAMm there, we can actually use multiple models in here. So sort of when we're looking at the sizes of the around the 35e or around the 27 billion dense models, those ones we can actually have more than one of those models actually loaded and do it. So you can see here I've asked it to basically find some latest AI news. It's finding sure enough about the new GPT 5.6 launch. It's talking about the new meta launch in here. It's gone and done a pretty good job at that. So, normally for Hermes, I have about 60 skills that I run with it.

[00:19:27]If I bring those in and run them in here and then run cron jobs and stuff along with it, you can have a very powerful local agent that's running 24/7 that's just sort of sitting there quietly that's using LM Studio for your models. It's not costing you anything except for electricity costs. you're not having to pay for your tokens. It can be chugging out millions of tokens a day and not be costing you anything except the electricity cost. And I think the big thing here is that going forward, we know that these models are getting better and better. If we just look at the Gemma 4 models, so I also have the Gemma 4 12B in there, which I play around with this as well. But if we look at the Gemma models, we know that each iteration, the sort of smaller models are getting as good as the previous bigger models. So I don't see that stopping anytime soon, especially as people are working out a whole bunch of new ways to get better at speculative decoding. Some of the things we've seen come out of DeepSeek and other labs means that this has got a two TB drive.

[00:20:33]I can have a bunch of these models just sitting on there with LM Studio and then bring them up as I need them. And you can see because I've got multiple models, I can run an embedding model in here. I can run a whole bunch of these different models. I generally leave LM Studio running 24/7 in there, except if I wanted to do something big with Comfy UI or something like that. But this is definitely an alternative that you've got now if you want to run these kind of things locally. And this is where the Halo really does kind of shine at this kind of stuff for just making it super simple, easy to get up, easy to get running. And I think this is one of the places where this hardware is going to do really well for people who are enthusiasts that want to have the model running locally on their desk, but they're not hardcore into running Transformers library or the VLM library.

[00:21:25]That said, VLM is on here and you can run models from that as well if you want to do that. All right, so running their playbook for using Unsloth is pretty straightforward. It will just download.

[00:21:38]They've got the environment set up there. You can basically then download your model. You can then kick off the training. Once the weights are down and stuff like that, you'll see it's basically downloaded both a data set and the weights here and it's chugging along and doing 50 steps of training here.

[00:21:54]Now, in this case, the model's still pretty small. It's the Gemma 4B. If you wanted to, you could certainly go for a bigger batch size and just increase the gradient accumulations. Remember, gradient accumulation, if you don't know what that is, is that for each forward pass, we can just accumulate the gradients, but not process them until we've got, say, perhaps four steps or eight steps. And if it was four steps and we're doing batches of four, then we're just calculating the gradients every 16. So it's like we're doing a batch of 16. It's just that we're doing it in four parts before we actually calculate the gradients and do the back pass. There you can see that it finished doing the training reasonably quickly and then basically merged the Laura weights into the actual model in 16 bit here. Now of course you're not going to get the same performance here that you're going to from a node of eight GPUs that are running and being able to load really big models and fine-tune them. But I think pretty much everything that you could do in a Google Collab, you would be able to do here. Obviously, if you're going to be fine-tuning bigger models, you're going to need more RAM.

[00:22:59]You're going to need to basically be on top of that and be paying attention to how you're actually sharing the RAM. And probably for the sort of mediumsiz models, if you wanted to fine-tune those, you're going to need to use that gradient accumulation with smaller batch sizes to be able to get it done. Maybe an overnight job that you turn on and just leave it. And just on that point, I've had the machine turned on now for three days straight. It's very good. It doesn't seem to overheat. It's actually got some fans in it where you will hear the fans spin up when you're doing very high GPU loads, but it certainly seems to be able to take those sort of overnight training jobs and overnight inference jobs. Okay, so just to wrap up, who do I think this box is for? If you want to run really big openw weight models, especially things beyond sort of 40B, and you want to run them locally and privately on your own desk, then this is genuinely a practical way to do that. And with the way that memory prices are going lately, the price that they've got on this is kind of to be expected and is pretty aggressive for what you're actually getting here. If your workloads are more about speed and small models, then maybe a discrete GPU workstation like I showed last time actually does make sense. But don't forget allin this machine is actually a lot cheaper than the one that I showed you last time. I've said it many times before. I do think that local AI in many ways is going to be if not the future, certainly one of the clear futures. The models keep getting better and better.

[00:24:31]What was possible in the size of models that we could run on something like this last year now has just gotten so much better with the Quen 3.5, 3.6 family, with the Gemma 4 family, with a lot of the sort of interesting fine tunes that people are making on those base models.

[00:24:49]Anyway, I've put all the links in the description. Go and check it out. I think on AMD's site for this, they've got links to where you can buy them locally in the various countries, etc. And let me know in the comments what are the top priorities of things that you would run on 128 gigs of unified memory.

[00:25:06]I'm really curious to hear from people whether they want multiple models per some of those models being image models as well as LLMs or whether they just want to run one big LLM. So check out the links in the description and join the AMD developer community and start building with AI. As always, if you found the video useful, please click like and subscribe and I will talk to you in the next video. Bye for now.