This 744GB Model Shouldn't Fit on Your Laptop. It Does

[00:00:00]Can you fit a GLM 5.2 class model on consumer laptop?

[00:00:06]Well, this shouldn't be possible, but now thanks to this new project called Colibri, you can do this.

[00:00:13]And in this video, I'm going to explain how exactly does this work.

[00:00:17]Okay, but first we need to talk about memory wall. In order to load these models for fast inference, you need fast memory. Think about several H100s that you would need. Each one of them costs about 20 to 30,000 dollars.

[00:00:33]Now, even to serve this model in four bit, you will need multiple of these.

[00:00:38]Which essentially means you're renting compute from an inference provider.

[00:00:44]However, the trick is that you don't need to have the entire network in memory most of the time. And mostly the memory remains idle.

[00:00:54]Now, this is thanks to the mixture of experts architecture that most frontier models use today.

[00:01:03]So, in case of GLM 5.2, there are 256 experts per layer.

[00:01:09]However, for every token, you don't need all of them to be active. There is a router which picks top K for every token.

[00:01:19]Which means most of the network is not used at all.

[00:01:23]And essentially, that 744 billion total parameters squishes down to only 40 billion active parameters per token.

[00:01:32]Now, in order to understand how Colibri takes advantage of this, let's look at what happens when you move token to token across a sentence.

[00:01:43]The first token lights up one small subset of experts and ignores the rest.

[00:01:48]The next token lights up mostly different subset.

[00:01:52]There might be some overlap, but the sets uh keep shifting for every token.

[00:01:57]Now, as you can see, most of the network is going to remain idle for uh the tokens that are coming in.

[00:02:04]Now, each of these expert is about 9 MB of weights, which raises the obvious question, if most of them are idle, do they even need to be in the working memory or RAM?

[00:02:18]Okay, so if you think about it, we can split the model into two different parts. One is going to be the hot part and the other going is going to be the cold part.

[00:02:28]So, the hot part is the dense stuff. Uh think about attention, shared experts, embeddings, which are small and used on every single token.

[00:02:38]Now, the cold part is that giant bank of routed experts, which are enormous in size, but they're going to be mostly idle at any instant, as we saw before.

[00:02:52]Now, here's an interesting fact about GN5.2.

[00:02:56]Only about GB of routed experts actually change from one token to the next.

[00:03:02]So, theoretically, you just need 11 GB that changes from token to token.

[00:03:09]Okay, so if you think about this, you need to just keep the hot part close to yourself, and then the cold part is going to be fetched as it's needed on demand.

[00:03:19]Okay, so the trick that this Kelby project uses is that it divides the memory into three different tiers.

[00:03:27]Tier one is RAM. Uh it's the small dense part of the network that lives there.

[00:03:34]Uh and you need about 9.9 GB in total, which needs to be always ready.

[00:03:41]Now, tier two is the NVMe SSD, where the full 370 GB expert bank are going is going to reside on disk.

[00:03:53]Now, tier three is optional. It's a hot slice of GPU memory if you have a GPU available. Otherwise, this could theoretically work on a CPU as well.

[00:04:04]Okay, so the simple idea is that you have a workbench and then you have the warehouse as your disk. Workbench is your working memory.

[00:04:14]Now, you are making a trade-off of speed in order to run this model. And this is a big discussion right now. However, this idea is very similar to the Dwarf Star 4 video for DeepSeek models that we looked at before. I highly recommend to watch that video after this.

[00:04:33]Okay, one more important detail. How the 744 billion parameter fits into that 370 GB in the first place.

[00:04:42]So, we are actually using 4-bit quantization instead of the full precision.

[00:04:49]Now, this is different than the 2-bit quantization that is used by Dwarf Star. It shrinks the model about 4x and you pay for a few point reduction in accuracy. So, it's lossy.

[00:05:05]However, it does preserve the performance.

[00:05:08]Now, thanks to this quantization, we can actually keep the experts in the memory.

[00:05:15]All right, but there are some things that you actually need to keep it in 8-bits of For example, if you're using the multi-token prediction head, that needs to be in 8-bit, otherwise it's going to collapse completely and you're not going to get that performance.

[00:05:29]All right, so this is a very important distinction to keep. Otherwise, if you're running this model in 16-bits full precision, you're going to be needing 1.5 TB, but thanks to this 4-bit quantization, we can reduce it to manage size that can run on an SSD.

[00:05:49]Okay, but that's only the part of the story. Okay, so you are able to load the model. However, you need to think about KV cache because that enables how much uh context window you can actually use.

[00:06:02]Now, as the model reads your context, it stores the KV cache that grows with every token it sees.

[00:06:09]Now, that KV cache is going to grow as you're introducing more and more tokens.

[00:06:14]So, GLM's 5.2 multi-latent attention compresses that cache down to 576 numbers per token instead of 32,786.

[00:06:27]That's roughly uh 57 times smaller, which is what keeps a million token context affordable on 25 GB.

[00:06:36]Now, the trick is that Colibri writes the compressed cache to disk. So, reopening a chat is warm instead of starting cold. Okay, so let's actually talk about Colibri, which is Italian for hummingbird. It's a pretty nice name.

[00:06:51]So, the whole thing is one uh C file, which is about 2,400 lines.

[00:06:58]Uh it's very similar to the Dwarf Star um inference engine that we looked at in the previous video.

[00:07:05]Now, the interesting thing is there are zero dependencies. Uh you don't need any Python uh runtime environment, and it doesn't even need a GPU or CUDA. It's just C and the operating system that you are running this on.

[00:07:18]And that's kind of the beauty, right?

[00:07:19]So, you have that 370 GB model file that lives on your disk, and this engine is going to load the appropriate parts into memory.

[00:07:31]So, this is going to only fetch that 11 GB of expert that changes.

[00:07:36]Now, keep in mind, the goal of this is not to to these models at high infant speed.

[00:07:44]It's just an open source project that actually shows you that you can run these humongous model on consumer hardware.

[00:07:52]Now, this part is extremely important because there is a lot of interest in this area. So, for example, today Prism ML announced uh that they can run that 27 billion parameter model on a phone, which wasn't possible before.

[00:08:07]Now, it may not give you the same level of performance, but just having this direction is critical.

[00:08:13]Which means in future we're going to have much better algorithms which are going to run these bigger models at much better performance.

[00:08:20]All right. So, here's how this whole thing actually works.

[00:08:24]So, the router names the expert it wants and the memory map reads those weights straight off the NVMe or your SSD into memory.

[00:08:35]There's also a caching mechanism which basically caches the recently used experts. Uh so, repeats don't hit the disk twice.

[00:08:45]But, the main problem is that it reads about 11 GB from the disk per token. So, this is going to be excruciatingly slow.

[00:08:57]You cannot interact with it because the model is going to take forever to generate responses. So, this is essentially useful as a proof of concept or for running async tasks that you don't care about in terms of speed.

[00:09:13]Okay, but it doesn't mean that it's going to always remain slow. The longer you use this, the faster this is going to get.

[00:09:22]Now, the idea is that they use a file that quietly records which experts you actually uh your actual prompts are getting routed to.

[00:09:32]And then it pins the hottest ones into any uh spare RAM you have got.

[00:09:39]So, they never touch the disk again.

[00:09:42]So, in theory, the more you use it, the more it's going to learn about the experts that it needs and that it's going to keep them resident.

[00:09:52]And you're going to have fewer and fewer 1 GB read per token. Now, this is going to work provided you have enough VRAM or uh RAM in general.

[00:10:03]So, there are some reports where you can get about one token per second on M5 Max.

[00:10:10]So, it's kind of really nice, right? So, the hummingbird that learns your flowers and comes back to them first when you need them.

[00:10:18]Okay. So, there are some other tricks as well which can actually improve the generation speed. So, if every uh forward pass already costs 11 GB of disk, the fix is to emit more than one token per pass. Uh this is basically the idea of speculative decoding we looked at it.

[00:10:39]Uh there are is a very interesting approach from DeepSeek DeepSpark.

[00:10:44]I created a video on it recently, so I'll link to that. Now, the idea is that you can draft several tokens cheaply, then verify all of them in one expensive pass.

[00:10:54]So, GLM 5.2 actually ships natively uh with a head for this.

[00:11:00]And it can land you 2.2 to 2.8 tokens per forward pass.

[00:11:07]But again, there is one caveat that the head has to be in 8-bit.

[00:11:12]If you want to use it in 4-bit, the acceptance collapses from 39 to 60% down to almost zero. So, one thing to be clear, you're not going to be doing inference for multiple That's out of the question. However, just to be able to run a frontier level model which outperforms any open-weight model uh it's actually on par with some of the closed frontier models. It's pretty incredible.

[00:11:38]Now again, you're not going to be breaking any speed records, but keep in mind you can get a sense of how a frontier model running on consumer hardware feels like.

[00:11:51]Now, if you are brave enough to try this, make sure that you have more than 25 GB of VRAM with a fast SSD disk and you can download the weights from Hugging Face.

[00:12:03]Now, there are some concerns about its impact on your SSD. The main question is going to be can you wait for the response generation speed and how long it's going to take.

[00:12:16]Now, I personally really like these ideas, these novel techniques in order to run these frontier models on consumer hardware. In some cases, they may not be practical right now, but this is a really great research direction.

[00:12:32]Anyways, do let me know if you are going to be trying this on your own hardware.

[00:12:37]Link to everything is going to be in the video description. Anyways, I hope you found this video useful. Thanks for watching. And as always, see you in the next one.