Can a 3.5GB model replace my 35B daily driver? (Bonsai 27B)

[00:00:00]This 3.5 gig model claims it can do the same job as a 21 gig daily driver. I don't believe it yet. So, I gave both of them the same broken server. Same 20step job. Both of them found it. Both of them fixed the same line. But look at the clock. The 21 gig model finished in under a minute. The 3.5 gig one took over two. That's six times the file and it's the faster one. That's backwards.

[00:00:27]And how about the design skills? Can it write good web pages and code applications? So, which one earns the slot on your box? I ran about 30 of these to find out. So, Prism ML took Quent's 27B model and made a bonsai version of it. Two builds. There's a turnary one at 7 gigs and a binary one at 3 and 1/2. And just for scale, the original dense model at full precision is over 50 gigs. Most of the coverage skip this part. So let me be clear about it. This is not quantization. A normal Q4 quant takes a fully trained model and rounds the decimals down. You start at 16 bits. You squeeze it to four and you hope the quality holds. The model was finished before you touched it. Bonsai weights never had decimals in the first place. Every weight in the binary version is minus1 or + one. and the turnary one also gets a zero. You can't round your way to that after training.

[00:01:31]The model has to learn like that from day one. If you know QA quantization aware training, it's that same idea taken all the way down to one bit. How they actually trained it, they haven't said. It's proprietary. My guess QA plus the ideas from Microsoft's Bitnet paper, but that's my assumption, so don't quote me on it. And they claim it keeps around 90 to 95% of the original models quality which sounds like marketing except people are already running this thing on phones and it's holding up on support.

[00:02:08]Mac users are ahead for once. Metal is already merged into llama.cpp 2. CUDA is still an open pull request.

[00:02:17]So to run this on my card at all, I had to build llama.cpp CPP straight from that PR. Now, my daily driver is Quen 3.635b.

[00:02:28]The Q4 version takes around 21 gigs with the expert split between my GPU and system RAM. It sits close to full model quality and it handles most of my everyday work. Quen does have a better model, the dense 27B. I can't run that on my setup. And that's exactly the model Bonsai is compressed from. So yeah, for me this test is personal.

[00:02:53]Everything runs on my RTX 3060 and every model gets the same settings, same KV cache quant, same effort level. I also added one more contender, Gemma 412B.

[00:03:07]The reason is simple. It's the best dense model my rig can run fully on GPU without any offloading. And it weighs about 7 gigs, which is basically the same as the Turnary Bonsai. same size, same constraints, so the comparison is fair. The first thing I want to check is how well it can build UI. I usually build small web apps for myself, stuff that tracks things or turns some multi-step operation into one screen.

[00:03:35]And I want these to look good, not just some buttons on a screen. So the task is one landing page, a fake product, one self-contained HTML file, and one hard rule. No external requests of any kind.

[00:03:49]No CDN, no web fonts, no image links.

[00:03:52]Everything has to be in line. That rule is the whole test. If you let a model pull tailwind, you're testing whether it memorized class names. Take that away and it has to actually write CSS. I gave the same prompt to all four models. One shot each. No retries, no fixing.

[00:04:09]Remember the two pages from the start of the video? Time to answer that. One of them came from my 21 gig daily driver.

[00:04:16]The other one came from the 7 gig Turnary Bonsai. And I'll say the uncomfortable part first. I could not call it from across the room. Both of them gave me a proper dark landing page.

[00:04:29]Real layout, real spacing, hover states, the pricing cards, all of it. One caveat and it matters. The turnary one stalled near the end of its file and I had to nudge it to finish. Hold that thought because it comes back later in this video. Quick calibration point. I also ran the exact same prompt through a big Frontier model just as a reference so you can see what the ceiling looks like on this task. It's not part of the competition. It's the ruler I'm measuring with. And structurally, every single page passed the rules. All of them. Zero external requests. All of them survive a phone screen without breaking. Five for five. The compressed models did not lose on correctness. So, how close does local actually get? Let me put my daily driver right next to the reference. Of everything I ran on this machine, thee page is the one that lands nearest to it. The spacing is considered the sections have rhythm and it uses depth instead of stacking flat boxes. It is not at the same level, but it is clearly much closer. That's a model running on my own hardware holding its shape against something running in a data center. And that is as far as the reference goes because it was never in this race. The race is my 21 gig daily driver against a 7 gig bonsai. Now the other two. Gemma's page is clean, but it's thin. 12 kilobytes against everyone else's 30. Almost no graphics. One media query. It reads like a template someone abandoned halfway. And binary, the 3 and 1/2 gig one, kept making layout mistakes. In my best run, the terminal graphic sits on top of the headline and covers it. I ran it a few times. It stayed consistently behind thee and turnary on design. So on design, turnary and thee sit in the same class and the MOE edges it for me. Binary and Gemma are a tier below. Web pages are one thing though. Next, I put all four of them on a live server with a real fault and let them lose. Okay, so now the next thing I want to test is how well it figures things out. A lot of the time I need to debug something on my rig over SSH and I want my AI to dig into it, diagnose the problem, and come back to me with a fix or at least a report. So, I set up a sandbox environment, a 2 gig file system on my server, and I deliberately filled it to 91%. There's one fat debug log eating most of it.

[00:06:58]Three normal service logs and one trap.

[00:07:01]A 150meg cache blob that looks deletable but isn't the problem. The model gets SSH access and one instruction. Find what's eating the disc. Fix it. Prove you fixed it. And there's a script that resets the whole thing between every single run. So every model walks into the exact same crime scene. Same as the design test. I ran a Frontier model through it once as the reference. Four commands. Check the disk, find the file, delete it, prove it. That's the ceiling on this one. Then all four models, multiple runs each. And here's the thing, every single run came back clean.

[00:07:39]The 21 gige, the 7 gig Turner, the 3 and 1/2 gig binary, and Gemma. All of them found the right file. All of them left the decoy alone. All of them proved the space came back. The only difference was chattiness. The Frontier reference needed four commands. My local models needed six or seven. That's it. This test could not tell a 21 gig model from a 3 and 1/2 gig one. But inside those clean runs, one moment is worth slowing down for. Two models hit the exact same confusing situation. The numbers on the disc didn't add up with what they just deleted. Same discrepancy, same weight.

[00:08:21]That's weird spiral. Watch what each of them does with it. Gemma runs one more command, checks the folders, does the arithmetic out loud. 150 + 300 is 450, matches the disk. Done. It reconciled itself to the megabyte. The binary model in an earlier loser version of this test hit the same wall and went the other way. It invented an explanation, told me the file system doesn't persist, and reported a 100 megs reclaimed when the real number was 1.45 gigs, off by 14 times, stated with full confidence. Now, to be fair to it, when I tightened the prompt, it ran this clean three times in a row. So, the fix was partly on me, but that's the difference I care about. When these models get confused, one of them checks, the other one writes fiction. So the scoreboard after this test is boring. Everybody passed. I built this thing to expose the small models and it refused to expose anybody. Which means the test was too easy. Time to build a harder one. So here's the harder one.

[00:09:28]This time it's not one command away. I built a tiny production setup on the sandbox. two services, a gateway that takes requests on one port and a back end it forwards to and I sabotaged it.

[00:09:39]The gateway is pointed at a port where nothing is listening. So every health check comes back 502. Finding that takes real digging. The health check tells you it's broken. The log tells you which port it can't reach. The config tells you why. That's three hops before you even know what to fix. And then there's the trap. The gateway's config file gets regenerated from a template every time the service restarts. So if the model edits the obvious file, the config, everything turns green. Health check passes, looks fixed. Restart the service and it's broken again. The prompt explicitly says the fix has to survive a restart. Oh, and I left bait. A 400meg junk log sitting right there for any model still thinking about disk space.

[00:10:25]and a scary out of memory error in an old log that was already resolved.

[00:10:30]Neither has anything to do with the fault reference run first. The Frontier model did the whole thing in eight commands. It read everything before touching anything, backed up the template before editing it, restarted both services twice to prove the fix held, and then listed every file it had modified. Eight commands. That's the ceiling. Now my daily driver, the 21 gige ran this three times and it went three for three. Found the chain, edited the template, not the generated file, restarted, proved it around 23 commands run. One thing worth noting though, twice it hit a weird glitch in my own health check script. And instead of saying that's weird, it confidently invented an explanation for it. Wrong both times. Keep that in your pocket because it's a pattern with these models and it's not about size. The 7 gig turnary also 3 for three around 24 commands basically the same shape as a model three times its size and it produced my favorite diagnostic move of the whole project. In one run its very first probe hit a completely different service running on my box and got back a 200. Everything looks healthy. A lazy model declares victory right there and walks away. It didn't. It kept reading, caught that the real gateway lives on a different port, and then did something none of the others did. It curled both ports, proved the backend answers on one, and nothing answers on the other.

[00:11:58]It ran the experiment instead of trusting the paperwork. 7 gigs of weights doing that. I'll take it. And the 3 and 1/2 gig one is next because that's a different story. The 3 and 1/2 gig binary also went three for three. It cleared the trap every time. and its best run was genuinely tight. But watching it work was a different experience. One run took 44 commands. In the middle of fixing the config template, it botched a shell command and wiped the file, blanked it completely.

[00:12:28]Then spent three commands confused about why the file was empty, blamed the wrong tool, and rebuilt it from memory. Same run, it found some of my servers actual system processes in a process list and tried to kill them with minus 9. The only reason nothing died is that the sandbox user doesn't have the permissions. It got there every time, but the path was chaos. And remember that chaos because it shows up again when the files get longer. Put the finishers together. Reference eight commands. My local models 23 to 32. On the easy disk test, they were less than double the reference. On this one, they are three to four times over. The gap didn't grow with model size. It grew with task difficulty. a 21 gig model and a 7 gig model paid almost exactly the same price. And you've noticed I've only shown you three models. The fourth one never made it to the finish line at all.

[00:13:23]Not once cleanly at its own recommended settings, and it's not the one you'd guess. So, the dark pane, the model that never made it to the finish line, isn't the 3 and 1/2 gig one. It's Gemma, the 12 billion parameter model. the one I picked because it was the fairest comparison on this rig. And look at what it's doing. Because this isn't garbage output. Every one of those calls is perfectly formed. It just keeps making the same one over and over. It never advances the task. So, the job never finishes. The loop isn't in the text, it's in the decisions. And before anyone says it in the comments, yes, I tried to fix the sampling. Gemma's own model card says run it at temperature zero. That's where it looped first. So I added a repeat penalty and a presence penalty matched to what every other model in this test was running. Looped again.

[00:14:20]Then I started playing with the temperature myself. Bumped it up. Ran it again. One clean finish. Then it looped again on the very next run. Penalties are the textbook fix for repetition.

[00:14:31]When the textbook fix does nothing, it's not issue with the config. It's the model itself. So, hold these two facts together. The 8 gig model can't reliably finish the job. The three and a half gig one finishes it every time. Now, remember the turnary caveat from the design test. Here's where that lands.

[00:14:50]Turner stalled on that page every single run, always in the same place. A long block of SVG coordinates near the end of the file. Same disease as Gemma.

[00:15:00]Repetition under long output. But scope matters. For Turnary, it's one narrow trigger and everywhere else it ran clean. For Gemma, it's the whole long task category. And this is the one place in all my testing where compression actually showed its bill. Stability under long context. The 21 gige never wobbled. Turnary wobbled once in one spot. Binary starts looping on edits when the files get long. That ordering follows size. nothing else in this entire test did. Okay, now the question the whole video has been sitting on. Why is the 21 gig model more than twice as fast as a 7 gig model on the same card?

[00:15:44]Here's the generation speed at a realistic context depth. The 48 tokens a second, binary 35, turnary 22. The biggest file on the board is the fastest model on the board. The answer is in one letter pair on the model card, A3B. That means out of 35 billion parameters, only three billion actually fire for each token. It's a mixture of experts. For every single token, a router picks a couple of experts and the rest of the model just sits there. The bonsai models are dense, 27 billion parameters, and every single one of them works on every single token. So, here's the sentence that explains my whole benchmark.

[00:16:25]Quantization shrinks the file. It does not shrink the math. MOE shrinks the math. The MOE is doing 3 billion parameters worth of work per token while Turner is doing 27 9 times the compute every token forever. A smaller file was never going to win that race. Same logic explains the gap between the two Bonsai builds. Identical parameter count, but generation speed is about memory bandwidth. Binary moves 1.1 bits per weight. Turnary moves 1.7. Less data per token, more tokens per second.

[00:16:59]Compression does buy speed, but only against the same architecture. On the actual server repair task, that's the difference between thee finishing in under a minute and the bonsai models taking around two. Same fix, same quality. The build is entirely in time.

[00:17:16]But here's the thing that actually impressed me. Flip the metric. Stop asking who's fastest and ask how much intelligence you're getting per gigabyte. The Turnary model is not beating the 35b, but thee needs 21 gigs to do it and turnary is doing near the same job in 7. The intelligence density of that thing is huge. And binary at 3 and 1/2 gigs is even denser. And remember, this only works because it was trained this way. Take a normal model and squash it down to one or two bits after training and it falls apart. It can never stand shoulderto-shoulder with a Q4. These can. Now, before the verdict, one number needs correcting because the whole comparison turns on it. I can only run my 35BOE because my machine has a second tire of memory. The experts don't fit on my GPU, so 26 layers of them live in system RAM. On paper, the uses 10 1/2 gigs of VRAM. In reality, it's a 21 gig model and half of it is renting space in my RAM. So the real column isn't VRAM, it's total footprint. MOE 21 gigs, turnary under 9, binary 5.3. That's the real size of each of these things sitting on your machine.

[00:18:33]And here's where that matters. That trick only works because I have a second tire to spill into. On Apple silicon, there isn't one. A 16 gig MacBook has 16 gigs of unified memory, and that's the whole budget. Offloading experts to system RAM doesn't make sense there because the system RAM is the VRAM. The would need the full 21 gigs of that one pool. It simply doesn't go. And remember the phone from the start of this video.

[00:19:02]Same thing taken to the extreme. A phone is the purest single tire machine there is. Nobody is running a 21 gigo on a phone. It's not that it's too slow.

[00:19:12]There is nowhere to put it. That's why the model people are running on phones is this one. So when is this useful? If you're on a Mac or anything with one pool of memory, this binary or turnary model is a great choice. Near level capability in a footprint that actually fits and metal support is already merged. So it runs from mainline llama.cpp today. Same story if you've got an 8 to 12 gig GPU and not much system RAM. If you can't offload experts anywhere, the MOE trick is off the table for you. And a dense model that squeezes 27 billion parameters into a few gigs is exactly what you want. One caveat for the Mac crowd. All my speed numbers are from my 3060. Apple has no PCI penalty.

[00:19:59]So if anything, these models should do better there. But I haven't tested it.

[00:20:04]Floor not a promise. So, can a 3 and 1/2 gig model replace your 21 gig daily driver? For me, not yet. Quen 3.6 35B is still the best I've got. Two reasons.

[00:20:18]The MOE is twice as fast on real work and it never wobbled on long context where the Bonsai models each have a weak spot. That's the whole case. On quality, on the design test, on the server repairs, these models basically tied.

[00:20:33]But that's my box. Your answer depends on your memory and your own rig. If you've got a 12 gig card and spare system RAM to hold the experts, run the MOE. It's the best thing in this test.

[00:20:45]If you're on Apple Silicon or any machine with one pool of memory, the Turnary Bonsai is the pick and it works on a Mac today. And if you're on a 6 to8 gig card, the binary one is the only 27Bclass model that fits at all. Just keep it away from really long files. And here's what I'm actually looking forward to. This video was turnary versus the obvious next step is a turnary because these two tricks stack. MOE shrinks the math. Turner shrinks the file. Scale this training approach up to a big mixture of experts and something in the 250 billion parameter class could land around a 20 gig download. Split that between a 12 gig card and system RAM, and you're running Frontier Scale parameter counts on hardware people already own. Nobody has shipped that yet. The day someone does, this verdict flips, and I'll be first in line to test it. If you're sitting on a Mac or a small card, tell me what you get. See you on the next one.