Local LLM performance tuning

[00:00:00]In this channel, we've looked at models, mixture of experts versus dense models.

[00:00:04]And the trade-off has always been uh that thee models tend to be much faster, but the dense models tend to be more intelligent and more capable. One model in particular has really impressed me, and that is Quen 3.6 27B dense. This is a fantastic model to use as a code assistant. Uh but because it's a dense model, all 27 billion of its parameters have to fire every time you ask it a question or ask it to do something. And that tends to make it much slower than thee models. Uh, can we speed it up a little bit? Can we narrow the gap? I'm not going to pretend that we can make a dense model as fast as ane model, but we can narrow the gap in performance uh and uh tune it so that it's not quite as painfully slow. Let's take a look at this. [music] >> Host.

[00:00:50]>> All right, let's start with a baseline reading. I am going to load uh Quen 3.627B. This is a dense model. I'm going to load it at Q4 and I'm going to load it entirely on one GPU. Uh let's just get a baseline reading with the model loaded on a single GPU. Start it up like this. Now, if we slide over to NV Top, we see that card one has not done anything. Uh card 2 is now sitting with uh 17 gigs of memory usage. That is that model loaded entirely onto that card. If we go to the web UI and ask my favorite testing question, what does the transient keyword do in Java uh and fire that off, it starts thinking. And if we go back and look at the terminal while it's working, we can see that we're getting about 21 tokens per second, give or take. This is a good baseline reading. I loaded it with all default settings. Uh no surprises, nothing unusual. Okay, so let's uh kill the server and let's let's start over. I'm going to this time load a slightly larger version of Quen.

[00:01:52]Uh this is Q6. So the model file is now about 22 23 GB instead of 17 GB. Same context window size. And I'm I'm still going to use default settings, but this time I'm not saying CUDA visible devices equals 1. I'm just going to start it up.

[00:02:05]And if we slide over to NV top now, we see that card one is using about 8 gigs per of of VRAM and card 2 is using just under 16 gigs of VRM. That's 24 gigs combined across the two cards. Great. So we go back and ask the same question and now we get 18 tokens per second. So we drop down by three or four tokens per second. Why? There's overhead. Well, I mean, first of all, it's a larger model.

[00:02:28]There's an extra 5 GB of stuff to work with. But also, we've got overhead of splitting that model across two cards.

[00:02:34]Uh the cards have to communicate with each other. And in the case of Saturn in the corner over there, it means they've got to communicate using a PCIe Gen 4x4 system bus. uh I refer you to uh this episode if you uh want more details on the hardware setup that we're using. But basically, we're limited to that PCIe bus. I don't have NV link set up between these two cards. So, they can only talk to each other through the PCIe bus. And there's overhead involved in that. Uh okay. Can we speed that up a little bit?

[00:02:59]Uh I didn't give Lama Server any specific instructions as to how to split that model. So, by default, it does a layer split. These models are divided into layers. And by default, Lama Server will just say, "Okay, fine. I'm going to take the first 20 layers and put it onto card one and the second 20 layers, put it onto card two, and the two cards can just deal with it. Uh there's other ways to do this. We can do a tensor split.

[00:03:19]This gets a little bit more involved.

[00:03:20]I'm not going to pretend I understand this at a very deep level, but it's a smarter way of splitting the model. Uh it involves a slightly different communication strategy between the two cards. Let's try this. So, let's start it up again. This time, I'm going to specify one extra option, and that is tensor split 1, 2. Uh 1, 2 ratio of how we're going to divvy up the model between these two cards. And because one of my cards has twice as much VRAM as the other one, I feel like 1, two is a pretty good ratio. So, let's start that up. Uh, if we slide over to NV Top, we see that the memory breakdown is more or less the same. Just under 8 gigs on card one, just under 16 gigs on card 2, 24 gigs total. Uh, great. And if we go back to the web UI, we should see the benefits of the tensor split versus the layer split. Right? So, if I start this process, we're going to see a huge improvement, right? Let's go and take a look at the speeds that we're getting.

[00:04:06]And we are getting exactly what we saw before, just over 18 tokens per second.

[00:04:10]But why? Why? I thought tensor split was way more efficient than layer split. Why why aren't we seeing that here? Well, it is more efficient, but only in certain use cases. The value of tensor split uh you see the benefit of that when you're doing what's called prompt prefill. This is ingesting data into the model as opposed to this where I'm sitting here asking it questions and getting it to answer me. This is generation. Uh uh the benefits of the tensor split are not really evident here. Uh so for this particular use case, a tensor split doesn't really get you anywhere. Uh the net effect is so small that uh it's basically uh it doesn't make any difference here. Okay. Uh this would speed it up if we were doing uh something heavily involving prompt prefill. I'm not going to get into that cuz my typical use case for this is is this. I want to get data out of the model. I don't care about putting data into the model. So what can we do to speed up generation? What can we do to make this thing output stuff more rapidly? Well, it turns out there's something called MTP, multi-token prediction. This is where there's an extra layer in the model. Uh the head, I don't know why they call it a head, but uh it guesses what the next couple of tokens are going to be at a time. And the model just has to say, "Yeah, I think that looks right. I'm going to use it." Or, "No, I think that's wrong. I'm going to compute the next token myself."

[00:05:18]And the guess is correct often enough that we see a huge speed bonus uh in generation speed. And if that sounds familiar to you, it's because it's a newer and better version of an older technique called speculative decoding.

[00:05:29]This is where you used to run a smaller model alongside a larger model and the small model was just there to guess what the next token is and the larger model would just approve or reject those guesses and it speeds it up a little bit. MTP is a better version of this.

[00:05:39]The head layer because it's part of the the larger model itself, it shares the same KV cache and it's uh it can do this much more efficiently. Uh speed increases of up to 50% or even more are possible with this. Now just to confuse the issue a little bit, sometimes this is an extra download. If we go to uh hugging face real quick, let's look at uh Gemma, the Gemma models. If we uh zoom in here, we see that there's an extra smaller download uh option here labeled MTP. This is the this is the actual head layer that you can download separately and you can tell Lama server server about it. Uh the Quen models are trained with the uh with the head layer already built into them. So, it's one download and you don't have to uh tell LMA server uh where the head layer is because it's right there in the in the ggo file that you downloaded. Uh there's pluses and minuses either way. With Gemma's approach, if you're not using MTP, you don't suffer the the VRAM penalty of loading that extra uh head layer into VRAM. With Quinn, uh because it's part of the G-O file, you're loading it into VRAM whether you're using MTP or not. So, you pay an extra VRAM penalty, but the convenience is that it's there if you decide to enable uh MTP. It's just an extra couple of parameters. Let's take a look at that.

[00:06:46]So it looks like this. Uh I'm going to we can ignore NP1 for now. We'll come back to that in a minute. We specify that we want to use MTP by uh giving it the draft MTP uh argument here. And then we can optionally give it a number uh between 1 and six. This is I think it's the number of tokens that it's going to guess at a time. You have to play with this number uh trial and error to figure out what works best for you. I find two is pretty good, but it might vary depending on your hardware. So play with that a little bit and just see what you get. Let's let's fire this up. If we slide over to NV Top, it looks like nothing has changed. We're still using just under eight gigs of VRM on card one and just under 16 on card two. It looks like the same exact thing. Uh but let's go back and ask our testing question again and let's see what happens. 36 33 35 36. We doubled it. It went from 18 to 36. It's a drastic improvement. uh which is great for generation speed. If this is what you're using the local LLM for to generate stuff, holy cow, this is a major speed up and you notice the benefits of this immediately. Uh a task that would have taken 10 minutes before will now take 5 minutes [laughter] with no other changes. All we did was add a couple of options and bam, we doubled our speed just like that. It's amazing.

[00:08:00][snorts] Uh so I know what you're thinking. You're sitting there thinking, what's the catch? You know, cuz there's always a catch. You can't just get a a double speed bonus for nothing, right? I mean, there's got to be a trade-off, right? Well, uh, yeah, there is. Let's go back to hugging face and let's look at there's a footnote here that's very easy to miss, but it's vitally important to understand this. Uh, we see right off the bat, MM Proj is not supported with MTP. What's MMR? A few episodes ago, we looked at image input. Uh, we looked at an application that can use an a local LLM to, uh, understand and tag images for us. That's great. That involves MM Proj. And as of the time I'm filming this, MM Proj is unfortunately not supported with MTP. It's one or the other. Uh so if you want to do image input, you have to disable MTP, which means you don't get that speed bonus.

[00:08:44]The other thing here that's easy to miss is values of NP greater than one are not supported with MTP. What is NP? NP is the number of things that this that the local LLM can do in parallel. Uh let's go back here. Let's actually start two tabs at the same time. I'm going to say, "What does the transient keyword do in Java?" And this tab is going to start answering that question. And over here, I'm going to say, "Hello, how are you?"

[00:09:12]And it's not answering me. Is it is it hung? Did it crash? What's going on?

[00:09:18]It's It's busy is what's going on. With NP of one, the LLM can only do one thing at a time. This This one is not crashed.

[00:09:25]It's not hung. It's just waiting. It's waiting for that other question to finish answering. uh the the LLM can't do more than one thing at a time when you've got NP set to one. Uh now if you're sitting at a web browser typing questions, maybe that's not a big deal, right? But uh if you're doing things agentically or in code uh where you're using multiple threads to hit the LLM at the same time, uh this is a problem because it means that only one of your threads can be serviced at a time and the others are just going to sit on the line waiting. It's still spinning here waiting uh for this first one to finish.

[00:09:53]Uh that's a problem. So the trade-off is I mean, yeah, great, you get a double speed bonus. That's amazing. But you lose the ability to understand images and you lose the ability for the LLM to do more than one thing at a time, which may or may not be a dealbreaker depending on what you're doing with it.

[00:10:07]Uh, now I should point out that uh, Lam.cpp is changing rapidly and it may be that they will eventually fix these two limitations. These are limitations right now. It doesn't mean there are always going to be limitations. So, by the time you're watching this video in the future, maybe they've already fixed that. That's great. But, as of the time that I'm filming this, uh, that's that's the trade-off. If you want that speed boost that comes with MTP, you got to give up image understanding and uh parallelism parallelism parallelogram. You got to give up the ability to do multiple things at the same time. So, uh there you go. It's a crash course on MTP. It's a huge speed boost for certain use cases if you're willing to live with the drawbacks of it. The most important thing is just to be aware of those drawbacks, uh what they are and uh what the trade-off is.

[00:10:46]And uh there you go. I hope that information is useful. We will see you next time.

[00:10:52]I'm not going to pretend that we can make a dense model as fast as ane model.

[00:10:55]I don't think that's possible. Uh but I shouldn't say that because who knows, right? If this sounds familiar to you, it's because it's a newer and better version of an older uh technique called what was it called? [laughter] I don't