How Quantization Shrinks Near-Frontier AI to Run on Hardware You Own

Added: 2026-06-04

[00:00:00]Julian Chaumond, co-founder of Hugging Face, one of the biggest AI platforms in the world, posted this in April 2026. A photo from a plane, laptop open on the tray table. He was running a 27 billion parameter AI model right there at 30,000 ft. No cloud, no API, no Wi-Fi, just the model running locally on his MacBook.

[00:00:24]And his verdict, quote, "For non-trivial tasks, it feels very, very close to hitting the latest Opus in Claude code."

[00:00:33]Rewind 3 years, March 2023. GPT-4 just launched. The leaked specs, 1.8 trillion parameters, 120 layers, 16 expert modules.

[00:00:46]To run it, not train it, just answer your questions, OpenAI needed a cluster of 128 GPUs in a data center drawing kilowatts of power.

[00:00:56]Running a model like that yourself on your own machine was out of the question. How did we get from a server rack in a data center to a laptop on a plane?

[00:01:06]Part of it is that models got smarter, better architectures, better training data.

[00:01:11]A 27 billion parameter model in 2026 is better than that 1.8 trillion parameter GPT-4 from 2023 at a fraction of the size.

[00:01:21]But even a smaller, smarter model is still too big for most consumer hardware. So, there's a second wind stacked on top, quantization, squeezing the model down until it fits on hardware you can actually buy.

[00:01:34]That's what we're going to talk about today. Every parameter in an AI model is a number stored at 16-bit precision, that's the standard, each number takes two bytes.

[00:01:45]That 27 billion parameter model Julian was running at full precision, about 54 GB just for the weights.

[00:01:53]That's more memory than almost any consumer machine has.

[00:01:57]There's a community on Reddit called local llama, a quarter million people all trying to run AI models on their own hardware.

[00:02:05]The all-time top post, 3,400 upvotes, says it all.

[00:02:10]Quote, "Enough already. If I can't run it on my 3090, I don't want it."

[00:02:15]That's a graphics card with 24 GB of memory. If it doesn't fit, it doesn't matter how good it is. So, you shrink it. The trick is simple. Quantization stores each of those numbers in fewer bits. 16 bits each becomes eight, or four, or two.

[00:02:32]At four bits, that 54 GB model drops to about 14, small enough for a graphics card. At two bits, under eight, small enough for a phone. Of course, you don't get that for free. The simplest way to do it just rounds every number to the nearest value on a coarser grid, like rounding 3.14159 to 3.1, or just to three.

[00:02:55]The fewer bits, the coarser the rounding, the more quality you lose.

[00:02:59]At eight bits, the loss is tiny. At four bits, it's noticeable, but the model still works.

[00:03:05]Push to two bits, though, and naive rounding falls off a cliff. Two bits is four possible values. Every weight, every one of those billions of numbers, snapped to one of four options. The model starts to come apart, hallucinations spike, it loses the thread mid-sentence. So, the real game isn't just shrinking the model, it's getting more quality out of every bit.

[00:03:31]Keep more of the model intact at the same size, and you can push to smaller sizes before it breaks. And naive rounding turns out to be crude in a few specific ways. Fix each one, and you claw back quality. Three ideas did exactly that. First problem, when you round every weight against the same ruler, you waste precision.

[00:03:53]A model is full of numbers at wildly different scales. A stretch of tiny values here, a few big ones there.

[00:04:00]Force them all onto one grid, and the small values get crushed into noise.

[00:04:05]The fix, don't use one ruler. Chop the weights into small blocks, say 32 at a time, and give each block its own scale.

[00:04:14]Now, the grid adapts to whatever range of values is actually sitting in that block. Then go further. Blocks inside blocks. A super block of 256 weights subdivided into smaller ones, each with its own scale. And those scales themselves stored compactly.

[00:04:34]Scales of scales. And one more refinement. Not all layers in a model are equally sensitive. Attention layers, the parts that decide which words relate to which other words, are fragile. Mess with them, and the model loses coherence fast. Feed-forward layers are more forgiving. So, mix precision. Spend six bits on the fragile layers, four on the rest. That nudges the average a little above four bits, but the quality where it counts jumps. The result is called K quants. The popular one, Q4KM, lands around 4.9 bits per weight on average, depending on the model. So, a touch bigger than naive four bit, but with far more of the model preserved.

[00:05:19]That's why it became the community default. When someone says they're running a four-bit model, this is almost always what they mean, not the naive version. Second problem. Push down to two or three bits per weight, and even local rulers can't save you. At two bits, you've got four points to snap to.

[00:05:37]Four. No amount of clever scaling fixes a grid that coarse.

[00:05:42]So, different approach. Stop rounding each weight on its own. Group weights, four or eight at a time, and look them up in a code book, a precomputed table of good weight combinations derived from sphere packing math, the kind that shows up in error correcting codes and crystal structures. Think of it like this, instead of saying each pixel is this exact shade of red, you say this block of pixels looks most like entry 47 in my dictionary. You lose a little, but a single number describes the whole block.

[00:06:14]Same idea applied to model weights.

[00:06:16]These are called I-Quants. This is what made two-bit quantization not garbage.

[00:06:22]At 2.06 bits per weight, a 70-billion-parameter model drops to about 18 GB, laptop territory, and it actually works. Not perfectly, but useably. Third problem, and it's the big one. Everything so far still treats every weight in a layer as equally worth preserving. But they're not. Some weights are critical. Small errors there ripple through the whole output. Others barely matter. So, which ones do you protect? The answer is called the importance matrix.

[00:06:56]Before you quantize, run the full precision model on some sample text.

[00:07:01]Watch which internal pathways fire the hardest, which channels consistently have large activations. Those connect to the weights you can't afford to round carelessly. And measuring that is cheap.

[00:07:13]You just run the text through once and tally up how strong each channel gets, a few minutes on an ordinary processor.

[00:07:20]Then you hand those scores to the quantizer. Important weight? Round it gently. Throw away weight? Round it hard. The model's the same size, the precision just goes where it earns its keep. At four-bit, this barely matters.

[00:07:34]The rounding's already fine. But at two-bit, it's the difference between a usable model and garbage. The harder you compress, the more it matters to know what to protect. And here's the kicker.

[00:07:47]The big research labs have heavier methods, GPTQ, AWQ, that crunch through far more computation and need expensive GPUs running for hours.

[00:07:58]The importance matrix takes a shortcut and skips most of that work.

[00:08:02]You'd expect it to give up quality in exchange. It doesn't. In head-to-head tests, it comes out essentially tied. In 10 minutes, on a laptop. All three ideas, the block structure, the code books, the importance matrix, came from the same person, Evan Kavrakov. A physicist who works in radiation therapy and Monte Carlo simulation.

[00:08:25]He showed up in the llama.cpp repo in mid-2023 and over about eight months built the entire quantization stack the local AI community runs on today.

[00:08:35]When someone asked him about publishing papers on any of this, he said, quote, there are no papers on K or I quants because I don't like writing papers.

[00:08:44]Combined with me enjoying the luxury of not needing another paper on my CV, I see no reason to advertise on archive.

[00:08:50]And then he added, not publishing lets other researchers ignore his work, which makes their methods look better than they actually are.

[00:08:58]Quote, so in short, a win-win. There's a running joke in the local AI community.

[00:09:04]Every time a new model drops, the first comment is always the same, when GGUS?

[00:09:10]When's the quantized version coming?

[00:09:12]Because that's the version people can actually run.

[00:09:15]And increasingly, that takes surprisingly little hardware. Someone got a 26 billion parameter Google model working on a 10-year-old server CPU, a 2016 Xeon, 200 bucks on eBay.

[00:09:29]The floor for good enough local AI keeps dropping. And it's not just small models. Remember two-bit, the setting that should be hopeless?

[00:09:39]We covered DeepSeek V4 in another video, a near-frontier model, 284 billion parameters that Salvatore Sanfilippo got running on a Mac at right around two bits.

[00:09:51]A lot of people were skeptical. Two bits for a model this good? No way. And they'd be right about naive two-bit, the kind that falls apart.

[00:10:02]But this isn't naive two-bit. It's the codebook trick for the bulk of the weights with the most important parts kept at much higher precision. That's what makes it hold together. The whole thing collapses to 81 GB and runs on a MacBook Pro around 30 tokens a second.

[00:10:20]It handles coding agents, calls tools reliably, lands close to today's best models.

[00:10:26]Think back to where we started, GPT-4 3 years ago, the frontier of the whole field, a rack of data center GPUs just to answer one question.

[00:10:36]This is better than that, running on a laptop.

[00:10:40]That's the compression. Underneath all of this is one simple fight. Fitting a model into the memory you actually have.

[00:10:49]That's the whole job of compression.

[00:10:51]And lately, that fight has been getting harder. You'd expect memory to keep getting cheaper. It basically always has.

[00:10:59]Lately, it's gone the other way.

[00:11:02]One industry measure, the cost per gigabit, jumped from 43 cents to $2.39 in 6 months, more than five times in half a year. The reason is structural.

[00:11:16]AI data centers are consuming a staggering share of the world's memory supply.

[00:11:20]A single Nvidia AI server rack uses about 37 TB of fast memory. That's the equivalent of a thousand laptops.

[00:11:29]One project, OpenAI Stargate, signed contracts for up to 900,000 memory wafers a month. That's 40% of total global production.

[00:11:39]For one project. Micron, one of the three companies that makes basically all the world's memory chips, shut down its entire consumer brand.

[00:11:49]Crucial, the RAM you'd buy for your PC, gone. Redirected everything to enterprise and AI.

[00:11:56]Their CEO said it's the most significant gap between supply and demand they've seen in 25 years. Analysts say 2028 at the earliest before prices normalize. SK Hynix's chairman said 2030.

[00:12:11]New factories take years and billions of dollars to build.

[00:12:15]Samsung's next major fab won't hit mass production until 2028.

[00:12:19]So, the usual escape hatch, wait a year, get more memory for the same money, isn't coming anytime soon. But, hardware was only ever half the story.

[00:12:29]The other half, how cleverly you compress, is exactly where the progress has been.

[00:12:34]The gap between what needs a data center and what runs on your desk keeps closing. And not because the chips got cheaper, it's because people keep finding smarter ways to pack more model into less memory.

[00:12:47]Whatever needs a high-end machine today, give it a year or two. A few things we brushed past here could each be their own episode. How models got so efficient in the first place, mixture of experts, where a model with hundreds of billions of parameters only wakes up a sliver of itself for any given word. How to actually run one yourself, the tools and file formats that turn a download into a working model on your laptop. Or, if you want to go deeper and nerdier, the importance matrix. The math behind it is a strange century-old geometry problem, and there's a real fight over what data you even feed it, because near-random gibberish sometimes beats clean Wikipedia. If one of those is the one you want next, tell me.