Gemma 4's Big Update in 4-Bit QAT — Low VRAM, Local

[00:00:01]Google shipped one of the biggest updates Gemma 4 has ever had. 11 fixes, tool calling, chat formatting, thinking, vision, and so much more. Same model names, same weights, but everything underneath got fixed. We already covered the full precision model yesterday where we did the full hands-on. But in this video I am going to use this 4bit quad builds from Unsllo 6.7 GB on disk. The full precision version is around 24. Now this way the mortals like us can run it on commodity hardware. Same model, same fixes, quarter the memory. This runs on one card. Let's get right into this. I'm going to use this Ubuntu system. I have one GPU card. Nvidia RTX6000 with 48 GB of VRAM. Let me download the model. And while the model downloads, here is a quick shout out from our friends at Orchestra who are holding this really good and nifty apps hackathon next week July 22nd to 29th.

[00:01:12]With Orchestra, you can now build real apps connected to your actual tools and data through MCP servers. Just describe what you want in chat and it generates the workflow app in full. I will also be taking part. You can even use Kimik K3 directly which we also have covered.

[00:01:30]They are giving away around $2250 US in prizes, $1,000 for the most useful app, $750 for the best weird app, and 500 plus swag for the community buzz winner.

[00:01:43]So check it out. I will drop the link in video's description.

[00:01:48]And while the model downloads, here is a very quick review of what actually changed. As I said, 11 fixes and they cluster into three groups. Tool calling, accuracy, consistency, and turn closure.

[00:02:02]Plus the assistant turn getting properly restored after a tool call. That's the one I'm testing live today. The green dots one. I will just show you up here as you can see. Then chat and formatting, null handling, missing run tags, duplicate tags, thinking content being retained properly, template regression reverted. That's six fixes on its own. And that's a plumbing that makes an agent loop actually hold together over multiple turns. We will also use it when we try it out with Hermes agent in this very video very shortly. And then speed and the vision, the blue dots. I already have tested those on my H100 card yesterday in this video.

[00:02:46]Flash Attention 4 is a big one there and again we tested it because this is hopper only. So on this A6000 today we don't get that flash attention before which is exactly why the rest of this list matters which I just showed you here. So uh because none of the other fixes need special hardware by the way.

[00:03:06]So we are good there and you can see that meanwhile the model has already been downloaded in context format and now let's load our model here. Let me paste the command.

[00:03:21]So this command here is loading our 4bit model plus its little MTP drafter with 64k context everything on GPU and this d- ginga is the one that matters without it lama.cpp CPP ignores tool definitions entirely and Gemma tool calls once come back as plain text instead of real calls. While it loads, it's very important to understand the concept of QA. In very simple words, normally you train a model at full precision, then squash it down to 4bit afterwards. The model never saw that coming, so quality drops. QAT does it differently. During training, it fakes the four bit rounding on every forward pass. So the model feels the damage while it's still learning and adapts its weights around it. By the time you actually quantise it, there is no surprise. It's already trained this way. That's how 12 billion goes from around 24 GB down to 6.72 with quality Google says stays close to the full precision model. So that is what is happening in QA.

[00:04:35]Meanwhile, the model is running with the KV cache. So let me show you the VM consumption. So consuming even with that 9.7 gig of VM, you reduce the KV cache, it is going to get reduced. What I would highly discourage is from the real world perspectives because for example if you're integrating it with Hermes agent that needs at least around 65k of context and this is where the model which is being served by llama.cpp here is now configured with hermes agent and we can see that the model is up and running. So what I'm going to do here, I'm going to check out this model on this pull stack real world app which is a C port freight tracker fast API SQLite full operations dashboard. It track vessels coming into a container terminal and which birth each one gets assigned to. And there is one bug in here a safety check that's supposed to stop oversized ships docking at births too shallow for them. So for example this MV coral runner I assign it to birth 4. So you see the birth force max draft is 9.5 m whereas this vessel is 9.8 m. So it shouldn't allow us to dock but you see it allowed us and if it happens in near world you know real world it will be crash the vessel would just run a ground it will be a chaos. So it's a very very critical issue which I'm going to ask this model to fix with the help of hermes agent and that will tell us if the quality is compromised or not.

[00:06:15]And this is the model. I'm going to give it this one goal. And you can see that there is no hint, no bug count, no file path. Just find every bug, fix it, and then sorry [clears throat] the start the server and prove it with um that it works or not. I'm going to run it. While it runs, let me take you to this diagram very quickly because we discussed these things. But I just want to give you an overview if you're wondering what that MTP drafter was all about. Normally the big model runs once per token. Three tokens three full passes slow with speculative decoding. A small draft model guesses three tokens ahead and the big model verifies all three in one single pass. The one it agrees with are kept. Wrong ones get thrown out and regenerated properly. And so is the case with MTP. the same idea but with no second model. The extra prediction heads are baked into the weights trained alongside the main model sharing the same hidden state. So you get three tokens out of one forward pass with nothing extra to download. That 254 mgabyte file which we pulled is just a drafter head and the result is roughly 20% more tokens per second. And not only these things plus all of these improvements have made this JMA 4 a real contender for production applications. I think they are going to fill in the gap left by the qu model which we are still waiting and that is why this is such a big deal.

[00:07:59]And our QA model is running and look at this reasoning block. It's reading the tool docs, catching its own mistakes and carrying that thinking forward turn after turn. That's a thinking fix on the list. Working amazing. And you can see that is now checking various tools and also fixing its own mistakes preparing terminals and tool calls are very intelligent, very grounded instead of just going bonkers. I think uh not only the full precision one as we saw this Q8 GGF I think amazing stuff and within a jify it has found the bug which is plus 0 plus 0.5 instead of minus.05 05 exactly the planted bug port 882 amazing so the [clears throat and cough] sorry the draft clearance check I think working superb let's check the application once it finishes all the tests because it is now just creating its own tests and checking it out shouldn't take too long now okay so all tests are done let's check the application Uh so let me just refresh this application.

[00:09:17]Let's select that vessel and now let's try to assign it to birth.

[00:09:24]And there you go. So you see that assign failed and vessel does not meet this clearance margin. So I can say that with certaintity that not only the speed has improved but also the quality hasn't been affected at all.

[00:09:43]And for the speed and speculative uh side of things, look at this Llama server logs. And this is what MTP actually is doing across the whole run.

[00:09:53]15 and a half thousand drafts. Nearly 85% of them accepted. 68% of every token it drafted was kept. That's 2/3s of the work done in the draft pass instead of a full forward forward pass each time.

[00:10:07]Also, check out this last line. chat format pack gema 4 that's a fixed chat template loading so as I said earlier same fixes Google shift now running 4bit on a single card it found the bug fixed it started the server proved its own work 6.7 GB no mega GPU no API key by the way it is now also supported in Olama now so you can just simply use that quad with Ola or LM Studio that's it let me know What do you think? I have recently started this free weekly AI newsletter which you can subscribe at fadmeda.com fadmed.substack.com.

[00:10:48]Plus, if you're looking to rent a GPU on very good price, you can find the link to Master Compute in video's description with a discount coupon code of 50% for range of GPUs.