These Lunatics Put a 27B Model on an iPhone

[00:00:00]Bonsai 27B needs 54 GB of memory to run at full precision. [music] This week, the absolute mad lads made it run on an iPhone. 27 billion parameters [music] in under 4 GB at 11 tokens a second, all without touching the internet. [music] It fits because Prism ML threw away 15 of the 16 bits in every single weight.

[00:00:24]What's left of each number is a plus or a minus sign, >> [music] >> and they claim it still keeps 90% of its benchmark scores. Is that a legit breakthrough or a flat-out lie? First, why 54 [music] GB? A model's knowledge lives in its weights, 27 billion numbers, and by default, each one is stored as a 16-bit float.

[00:00:47]2 bytes, 27 billion times over. Got it?

[00:00:51]Bonsai's base model is dense, meaning every single one of those weights works on every token it generates without shortcuts. [music] Phone is a harsher target than the spec sheet suggests. The iPhone it ran on has 12 GB of RAM, but iOS lets a single app claim only about six before the system kills it. 6 GB for everything, the weights, the conversation history, and the runtime. Usual fix is quantization, storing each weight with fewer bits. A standard 4-bit build of this exact model still weighs 17.6 GB, closer, but still three times over the ceiling. Getting a dense model this size through a 6 GB gate needed something far past the usual trim.

[00:01:39]What does one bit per weight even mean?

[00:01:42]The most upvoted confusion when this launched [music] was some version of a bit is a zero or a one. So, how is it holding a minus one?

[00:01:50]>> [music] >> Every weight in the original model is a precise number, something like [music] 0.0173.

[00:01:58]Bonsai one thing about it, the sign.

[00:02:01]Positive becomes +1, negative becomes -1, and the magnitude is gone. [music] To stop the math from turning to mush, the model stores one full precision scale for every group of 128 weights, a shared volume knob that says how loud this whole block of signs plays. [music] 128 1-bit signs plus 16-bit knob [music] works out to 1.125 bits per weight. That's the true number, and it matches the label, which is rarer than you'd think, [music] because quant labels lie. The community's favorite 4-bit build of this model is really 5.2 bits [music] per weight once you count the scales and the escape hatches. The popular 2-bit one is 2.8. Bonsai also ships a turnery sibling, -1, [music] 0, +1, at 1.71 bits. But the binary build is the headline act and the one that fits the phone. Round a finished model's weight down to bare signs, and it dies on the spot. Below three bits, conventional quantization stops degrading gracefully and collapses. Half the Hacker News thread assumed Prism ML dodged this by training a 1-bit model from scratch, the way Microsoft's BitNet does. They didn't. They took [music] Qwen 3.6 27B, an existing dense model from Alibaba, and retrained [music] it with the 1-bit constraint switched on inside the training loop, so the optimizer learns to route around the missing precision while it can still adjust everything else.

[00:03:36]Quantization-aware training, the model heals around its own wounds. Two more tricks on top of that. Long conversations normally [music] blow the memory budget a second time through the KV cache, the model's running memory of the context. [music] So, Bonsai compresses that to four bits as well. The full 262,000 token window fits in about 9 and 1/2 GB, and the runtime never unpacks the weights back to full precision. Custom kernels run the matrix math directly on the packed bits, >> [music] >> signs, volume knobs, and all. This last bit is important to understand for the love and hate of this model. Because the reviews are unhinged. One developer on Reddit said the one-bit [music] build passed his personal edge case tests and felt as good as the full model. Another said it couldn't write CSS without looping. A third asked it what Signals is and got a confident wrong answer.

[00:04:34][music] And someone on Android received 50 exclamation marks and nothing else.

[00:04:39]All of them were describing the real model. Three things decide which one you get. First, [music] the task. The damage from the missing bits is targeted. Math barely notices. 92 versus 95 for the original. Coding gives up seven points, >> [music] >> but agentic tool calling craters from 80 to 66, and instruction following falls 13 points.

[00:05:03]>> [music] >> Ask it to reason and it holds. Ask it to run a multi-step workflow and it walks into a doom loop, failing the same step the same way forever.

[00:05:13]>> [music] >> Second, language. English survives the compression best. Push it in German or Italian and the grammar starts crumbling, sometimes [music] mid-word.

[00:05:23]Third, the kernel trap. If your runtime lacks the packed [music] one-bit math path for your hardware, it unpacks every weight back to eight [music] bits before multiplying. You pay the full unpack cost and receive none of the speed.

[00:05:38]That's how people measured a one-bit model running slower than a four-bit one, and it's why a chunk of the day one its broken posts were really, my back end isn't supported [music] yet posts.

[00:05:50]About that 11 tokens a second, it's real and it's below the 15 to 20 that feels conversational. You sent this model off with a job and check back the way you'd treat a slow but capable intern. The battery number impressed me more than the speed.

[00:06:07]>> [music] >> 672 tokens per 1% of battery. Roughly 67,000 tokens on a full charge. If you manage your expectations, the use cases are clear. One commenter stood in a grocery store in the Alps, no signal, and had the model on his phone plan dinner and the shopping list. Offline, private, [music] running on hardware you already own. Add document analysis or translation on a plane and you have the honest pitch.

[00:06:38]>> [music] >> If you want this to be your next live coding assistant, be prepared to cry.

[00:06:43]The whole field slid down the precision axis [music] in the same week. On Sloth shipped NVFP4 quants of the very same quant model. Four bits, [music] two and a half times faster, aimed at big Blackwell GPUs with memory to spare. On a data center GPU, quantization buys you speed. On a phone, it buys you control.

[00:07:04]It's also a different escape hatch from the one we covered at the start of this week, where a giant mixture of experts model streams its experts off an SSD.

[00:07:14]Streaming changes where the weights live. Quantization changes what the weights are. Streaming [music] wins when the model is sparse and the disk is fast. Extreme quantization wins when the model is dense and the memory ceiling is a hard wall. A phone is the hardest wall there is.

[00:07:32]So, breakthrough or flat-out lie? The compression is real and the phone demo is measured on named hardware with battery numbers attached. The 90% claim is true as an average, and the average height is exactly where the missing 10% [music] lives in the gigantic workflows developers care about most. A year ago, one bit quants were a meme on these same forums. Today, they run 27 billion parameters [music] on a phone, and the argument has moved to a harder question.

[00:08:03]Once it fits, is it the best model you could run in those 4 GB? Smaller model at 4-bit sometimes wins that fight, [music] and nobody has settled it yet.

[00:08:13]It's clear that one bit first started as a punchline. [music] After this week, it's a roadmap.