TurboQuant + DFlash: Supercharge Local LLM Speed

Added: 2026-05-12

[00:00:01]Turboquant from Google now works with D Flash speculative engine.

[00:00:07]And that is what exactly we are going to show in this video hands-on. This is Fahaat Mirza and I welcome you to the channel.

[00:00:14]Google research published something very important last month. A compression algorithm called Turboquant that can shrink the memory a language model uses during inference by six to 10 times with essentially zero quality loss. No retraining, no fine-tuning, no accuracy trade-off, just mathematics.

[00:00:35]Specifically, a two-stage approach which we also covered in great detail hands-on in these videos with many tools out there.

[00:00:45]Now, this new tool by Google works on two-stage approach using polar coordinate transformation and a single bit error correction.

[00:00:54]A trick called as QGL. I'm not going to go into that detail in this video because already covered in this video.

[00:01:01]Now, we are going to integrate it with Loose D Flash.

[00:01:05]Loose team has been building something that the local AI community has been paying close attention to, D Flash. A handwritten C++ and CUDA inference engine for various models that uses speculative decoding to run at two times auto-regressive speed on a single consumer GPU. Again, we have covered this D Flash with various variants and concepts and angles in heaps of videos in the last two to three days. No VLLM needed, no llama.cpp runtime, just raw kernel code talking directly to the GPU.

[00:01:40]Now, this is where it gets interesting.

[00:01:42]The Loose team did not just build a fast inference engine, they also built their own native C++ implementation of this Turboquant directly inside D Flash. They They it TQ3_0 which means 3.5 bits per value, 9.7 times smaller than full precision FP16 KV cache.

[00:02:06]In simple words, what that means in practice is that a context window of 128,000 tokens fits on a single 24 GPU.

[00:02:15]Without it, you are stuck at around 16 to 30,000 tokens on the same hardware.

[00:02:20]The same model, same GPU, same boundary, two environment variable is the entire difference between these two numbers.

[00:02:27]Let's get into installing and running this demo. I'm going to use this Ubuntu system. My GPU card is an Nvidia RTX A6000 with 48 GB of VRAM. Let me quickly create a new virtual environment with conda. It's always a good idea to do that because it just keeps things easy and separate.

[00:02:48]Let's wait for it to get installed.

[00:02:52]And that should be done any second. And by the way, if you're looking to rent the GPU on very good price, you can find the link to my Sky Compute in video's description with a discount coupon code of 50% for a range of GPUs.

[00:03:05]And now we need to install all of these prerequisites. This is going to take a bit of a time, so let's run this.

[00:03:15]Everything is installed. Let's now get clone the repo of this Loosely Flash.

[00:03:21]That part is done. Now let's build this from the root of the repo.

[00:03:27]That part is done. Now let me quickly download both of these models.

[00:03:34]While it downloads the model very quickly, why we are using two models?

[00:03:38]Well, Loosely Flash uses two models working together.

[00:03:42]First is a main model, our current 3.6 27 billion, which is the one that actually answers your questions and generates the final output. It is 16 GB and runs in Q4KM quantization to fit on any consumer GPU. The second is the draft model from model that was specifically trained to understand how this quant 3.627 billion things internally.

[00:04:09]So, before the big model generates each token, the draft model looks at the big model's hidden states. It's internal representation of what it has processed and proposes an entire block of tokens simultaneously using block diffusion.

[00:04:25]The big model then verifies all of those proposals in one single forward pass.

[00:04:30]If they are correct, they all get accepted and you get multiple tokens for the price of one verification step. That is the speculative decoding speed up.

[00:04:39]The draft model is not guessing randomly. It was trained specifically to match quant 3.627 billion tokens, which is why the acceptance rate is very high and the speed up is real. Let's go back.

[00:04:54]Our models are done and you can see we have two models. One is draft and the other one is the actual quant 3.6 model.

[00:05:02]And now is the time to test. So, what we are going to test here in the context of this Turbo Quant and Deep Flash.

[00:05:10]We are testing how Turbo Quant KV compression inside Deep Flash changes VRAM usage and context window size. The same model, same GPU, two environment variables, completely different capability.

[00:05:23]Let me first start this server.

[00:05:26]Uh and by server, I mean the loose Deep Flash server with standard FP16 KV cache, no compression, baseline measurement. So, all I'm doing is I'm just setting the budget to 222 context window and it is being served at port 8080. Let's run this.

[00:05:43]It is going to start it and then it is going to serve that model.

[00:05:48]First time takes bit of a time.

[00:05:51]Let's wait for it.

[00:05:54]And that server is started without any quantization, without any turbo quant as you can see here.

[00:06:01]Let me quickly show you the VRAM consumption. I'll just let it run.

[00:06:06]So, remember it is consuming around 19482 meg. We will also check out the KV cache soon. But remember this uh number that it is using this much.

[00:06:18]Okay, now let's do it maybe with the uh turbo quant.

[00:06:24]So, I'm just going to press control C here.

[00:06:26]The server is stopped.

[00:06:29]Let's go here.

[00:06:32]So, I'm just going to run this and I'm just going to double-check my VRAM.

[00:06:36]VRAM is back.

[00:06:38]Let's run this.

[00:06:40]And let's see how much this one takes with the quant compression of turbo quant.

[00:06:47]And you can see that we are using this parameter here for the turbo quant which I mentioned earlier.

[00:06:54]And now it is running with turbo quant as you can see. Let me show you the VRAM consumption.

[00:07:01]And there you go with the turbo quant, the usage of KV cache and the model is quite low as you can see here, which makes sense as per um the claims which they have made in their repo because the KV cache saving at 16K context is around I would say 400 to 600 MB, uh which is quite good. And the point here is that as a context increases, these savings would be more meaningful and that is the whole point here.

[00:07:33]And just to quickly elaborate it bit further, at short context like we have done here, the 16K [clears throat] Given cache itself is so tiny that even this 9.7 times compression saves only a few hundred megabytes, which is not that huge in the grand scheme of things. But as the context grows, the KV cache grows linearly. At 131K tokens, for example, it is eight times larger and without Turbo Quant, it would need maybe, you know, more than 50 gig just for the cache alone. And of course, on my GPU, it will give me an out of memory error.

[00:08:10]But with this Turbo Quant TQ3_0, that same context fits in under just 2 GB of KV cache. And that allocation making the impossible on the consumer hardware. Maybe I can run that command for you quickly just to show you that larger context.

[00:08:28]So, I am going to start that server again and look at this command. This is where again I'm using this Turbo Quant.

[00:08:36]And on both the key and value cache, I'm pushing the context to 131K token.

[00:08:41]That is 128,000 token on a single GPU.

[00:08:45]Without TQ30, this is going to fail. But of course, I don't have that much GPU as you just saw.

[00:08:52]But with it compressed KV cache, it should comfortably fit and we will also check our VRAM consumption in real time, too.

[00:09:01]Sorry, just make it here. I'll just move here. So, keep an eye on this VRAM consumption.

[00:09:07]Takes bit of a time, but we can't have everything.

[00:09:12]So, this is the whole point of this.

[00:09:17]So, it is starting the server, you can see. There at 15. 18.

[00:09:22]21. That's it. So, it is consuming just 20 just touch over 21 gig for this huge context window. How good is that?

[00:09:34]So, that's it. Let me know what do you think about this. Please follow me on X if you're looking for AI updates.

[00:09:40]And if you want to help out the channel, please become a member. Thank you for all the support.