Gemma 4 Just Got a Massive Update (Tested Live Locally)

[00:00:01]Gemmafour is a sweetheart, a darling.

[00:00:04]Gemmafour, Google's open multimodal family, five sizes up to 256k context.

[00:00:12]We covered this on release thoroughly.

[00:00:15]Today's not about the model. It's about what just got fixed under the hood.

[00:00:22]Five things changed this week just like few hours ago.

[00:00:26]Green dot on this HTML means I'm testing it live right now. Shortly, tool calling and vision.

[00:00:33]Hollow dot means it's Google's claim, not something I have verified myself.

[00:00:38]Flash attention for chat template, reduced laziness, all Hopper specific or unbenchmarked. So, I'm not going to pretend I tested them all, but look, if you are using that architecture, I'm more than sure you are going to enjoy these enhancements. This is Fahad Mirza, and I welcome you to the channel.

[00:01:00]So, for the purpose of this video, we are going to use this Ubuntu. This is my GPU card, Nvidia H100, the Hopper architecture.

[00:01:08]Fresh Ubuntu box. I'm going to serve vLLM.

[00:01:14]And this model, let me quickly run that.

[00:01:18]So, this is a serve command. This enables the Gemmafour tool parser, the [clears throat] reasoning parser, and a custom chat template vLLM ships specifically for this model. While it serves, let's talk what exactly has happened in detail without boring you much. So, here is actual breakdown. Google shipped five fixes just few hours ago, all on the same weights, no retrain, no new checkpoint. Two of them I'm testing live today, the tool calling parser, which had a real bug where malformed output would break under long context. You also mentioned it under my videos when we covered them last time.

[00:02:00]And the vision token budget, which controls how much detail Gemma pulls out of an image. The other three, flash attention for chat template smoothing, reduce laziness, I'm leaving as Google's claim, not mine. Flash attention for specifically only runs on Hopper GPUs.

[00:02:18]So, if you are on anything older, that one doesn't really apply to you at all.

[00:02:24]I'm using the Hopper, but the compilation and build of flash attention for takes long time, and I really don't have that much time today.

[00:02:34]But, I will definitely talk a bit more about it.

[00:02:37]Look at this. This is the E4 benchmark chart from Google's own materials, agentic and tool calling benchmarks BFCL, and few others. You can see that airline jumped almost eight points, telecom over six. These are Google's numbers, not something I ran myself, but they line up with what we are about to test live.

[00:02:58]And the second one, this image, I mean, is the one the official announcement straight [clears throat] from Google, and that is what looks quite promising.

[00:03:09]So, that is what we are going to test, but while it downloads the model, let me show you something very interesting.

[00:03:19]So, this is the model being downloaded.

[00:03:21]So, look at this vision token budget.

[00:03:25]So, here is what is happening here, and this is a hugging face demo.

[00:03:29]You tell Gemma how many tokens you want to spend on an image. That's your budget. So, for example, if I just say 70, you see this is the image it has just spent 70. Uh and you can see it landed on 384 by 384 pixel using just 64 actual tokens. Similarly, let me Sorry, let me click on 140.

[00:03:51]And it has improved, and this is how many tokens it has used. So, this algorithm takes your original photo and shrinks it down to fit that budget, chops it into smaller patches, then squishes group of these patches together into what's called as soft tokens.

[00:04:08]That's a compressed version of the model actually reads. Lower budget like this, faster and cheaper, but less detail.

[00:04:16]Let's say if I just bump it to up to 1120.

[00:04:19]And you can see that you get way more detail, which is better for OCR or reading fine text.

[00:04:26]I'll just scroll through.

[00:04:28]And this is the budget.

[00:04:30]Original, and then this is what it looks like.

[00:04:34]You see?

[00:04:35]So, this is the improvement they have done, and I think this is real economical, real good stuff.

[00:04:43]And while the model is served now, I just wanted to show you something very interesting which I also didn't know before.

[00:04:50]It is actually using flash attention for. I thought maybe I would have to recompile it, but that is not the case.

[00:04:56]Um so, FA4 or flash attention for is real and active on our H100.

[00:05:03]And I think this is quite good. Okay, let's now go and test it out.

[00:05:11]Model is being served, and let me show you the VRAM consumption at the moment.

[00:05:15]So, with KV cache it is using just over 77 gig of VRAM.

[00:05:22]So, for the testing, I'm using this script. This does two things. First, it checks if Gemma correctly hands off a weather question as a structured tool call, not just text. That is what was happening earlier in the previous version of the model.

[00:05:37]And then, this script feeds a fake result back and checks if the model actually uses it to answer, proving the fix holds across a real two-step exchange. And this is the whole code I'm going to use. Okay, so now let me go back to the terminal and run this.

[00:05:58]And there you go, very quick, very fast.

[00:06:01]As you can see, the fix is working. Turn one, content is empty, but tool {underscore} calls is populated. It correctly extracted get weather with Sydney as a structured call, not leaked as text. Turn two, we fed it a fake weather result, and it used that data to write a real answer. Clear, 21 degrees.

[00:06:23]That's a bug from the Hermes um thread, but they have patched it without any Hermes. Hermes is still lagging, by the way. But VLLM is working with the new model, and fixes are in. And I think now you can use tool call without any problem.

[00:06:41]So, that's it. That is uh what I wanted to cover, and honestly, with flash attention for support and a real tool calling fix, and this much control over vision tokens, I'm surprised Google didn't just call this Gemini 4.1. It's more than a patch note, but that's the kind of the point, I guess. Same weights, same download, real improvements underneath.

[00:07:04]If you're running Gemini 4 in any agentic setup, the tool calling fix alone is worth pulling today. And I know for a fact that lot of people, especially in the West, are using these Gemini 4 models, where they really don't want to go with any non-Western model, and they were really struggling to get these tool calling work with real-world use cases. I think this is a huge blessing for them.

[00:07:28]Um but, of course, the only uh problem here is that, especially with the flash attention, you have to go with the hopper, which is expensive, I know.

[00:07:37]But if you are serving this 31 billion parameter model in production, you really don't have any other option but to go with the 80 GB GPU card which I just showed you. Let me know in the comments if you are already using it and if you are looking to upgrade it very keen to know about your use case.

[00:07:57]Also, if you want to help out the channel, please become a member by either just going to this front page of the channel and becoming a member from this link or checking out the first pinned comment. Thank you for all the support.