Qwen3.6 27B (Pi-Reasoning GGUF) - Fine-Tuned for Local Heavy AI Agent

[00:00:01]This is a new fine-tune of Qwen 3.6 27 billion. One of the best model out there at the moment from the open source community.

[00:00:12]This new fine-tune model has been built by an independent researcher specially to make it better at agentic coding like reading files, running terminal commands, writing fixes, and checking its own work the way tools like Claude code or Codex or open claw do. This model was trained on real successful coding agent sessions including the step-by-step reasoning behind them, not just the final answers.

[00:00:41]We are going to install this locally. We are going to wire it into Hermit agent, and I'm going to do some real-world testing.

[00:00:48]The angle here is that I'm going to use a GGUF version, a quantized version in Q4KM, which I believe most of you can also run easily.

[00:00:59]This is Fahad Mirza, and I welcome you to the channel. We have been covering this Qwen model for quite some time, and it is still as fresh as ever. And you know, the fresh freshness of these models is not that prolonged. The shelf life is too low, but for Qwen 3.6 27 billion, it is still quite relevant in my humble opinion.

[00:01:23]Also, recently I have started a free weekly AI newsletter, which you can subscribe to from the home page of the channel. The tool which I'm going to use is this llama.cpp. It's already installed on my system. If you don't know how to get it installed, just search my channel.

[00:01:40]So, I'm going to use this Ubuntu system.

[00:01:42]I have one GPU card in video RTX A6000 with 48 GB of VRAM, and we will see how much VRAM it consumes one once we serve it with llama.cpp.

[00:01:54]First up, I'm going to download this Q4KM cord. There are also lower and higher cords, but I think this is a good balance.

[00:02:03]While it downloads, let me very very quickly give you a bit of an overview around speculative decoding and MTP because these are the two concepts which are used in this model.

[00:02:15]So, multi-token prediction or MTP is baked into the model itself, which lets it generate more than one word per thinking step instead of one at a time, which means it runs faster than a normal model of the same size.

[00:02:32]So, what exactly is this MTP? Let me tell you very quickly.

[00:02:38]Look, the normal model does one full pass through its network to produce one word, then does another full pass for the next word and so on.

[00:02:48]This model has extra extra prediction heads built directly into its weights, trained alongside the main model. So, in that same single pass, it also predicts the next two words ahead. No second model, no nothing extra to download. It's baked in, and that is a selling feature of this model.

[00:03:08]In the same way, speculative decoding is what actually turns those extra predictions into real speed.

[00:03:15]Normally, three words means three full passes through the model one at a time.

[00:03:21]Here, the model drafts several words ahead using those extra heads, then does a single pass to verify all of them at once.

[00:03:30]Whatever it got right, it keeps.

[00:03:32]Whatever it got wrong, it throws out and redoes from that point. Same output, fewer expensive passes, that's a speed up. So, hopefully now you understand the whole scenario what is happening with this model.

[00:03:46]Let's go back, and our model is downloaded. And by the way, if you're looking to rent a GPU on very good price, you can find the link to mass compute in video's description with a discount coupon code of 50% for range of GPUs.

[00:04:01]And now, let's serve this model. So, I'm just going to run this command model.cpp which is already installed. So, here dash m is pointing to the model file, and then spec type draft, you see m t p, and then we have draft and max. They turn on speculative decoding using the models built-in m t p heads.

[00:04:23]Drafting three tokens ahead per pass.

[00:04:26]NGL means that I'm offloading everything to our GPU. Dash f a and the cash type k v q 40 flags save v ram and speed up attention. We're setting the context window to 128k.

[00:04:39]Ninja enables proper chat tool formatting.

[00:04:42]And then we have some hyper parameter to control output randomness. Let's serve it.

[00:04:51]And the model is now being served. Let me quickly show you the v ram consumption.

[00:04:56]It is consuming just over 20 gig of v ram in this q4km.

[00:05:01]So, you should be able to you fit it in in your 24 GB v ram. You can also decrease the context length in order to decrease this v ram usage.

[00:05:13]So, for the first test, we are going to see if this model can troubleshoot.

[00:05:18]So, this is a full stack application with some back end and some front end where we have planted some bugs. And I already have started this application here. But when you try to access it in the browser, it gives us this error.

[00:05:33]So, what I'm going to do, I'm going to cancel this here.

[00:05:38]And I will ask the model. I'll just go one step back just to make sure the path is correct.

[00:05:44]By using Hermes agent's profile, I'm asking this model to fix it. The back-end is here, the front-end is here, and the model needs to use Hermes' agent in order to get this fixed and give me the correct running application. It is initializing the agent with Hermes.

[00:06:03]Let's wait for it to come back, and then we will go through it. It is going through the files, using the tools.

[00:06:12]Let's see if it is able to fix this application or not.

[00:06:19]And while it runs, let me show you very something very interesting. Look at this line. Draft acceptance.81818.

[00:06:28]That means out of every draft, the MTP heads guessed ahead, the big model agreed with 82% of them. That's a speed trick in action live. So, instead of generating one word per full pass, it's accepting multiple guessed words at once, and only redoing the ones it got wrong.

[00:06:49]It is still running. It is It has already fixed up all the files, and now just testing it out. Let it finish, and then we will test out our application.

[00:07:00]And the fix is done. Let me quickly refresh this browser.

[00:07:06]And there you go. Now, there is no error, and it has also produced all this data from the SQLite database from the local system.

[00:07:16]Looks pretty good to me.

[00:07:18]So, you see, just a quantized version with MTP has done really quite well.

[00:07:24]And now, let's switch it up. This time, no coding bug fix, just a pure creative coding test to see how it handles animation, timing, and visual polish from a text prompt alone.

[00:07:38]So, I'm asking it to create me a self-contained HTML file to animate a procedurally generated tree growing from the bottom center of the screen in real time over 15 seconds. And then there are few conditions like trunk should emerge, then go upwards, some branches, some stem leaf, and all that stuff.

[00:07:59]Surprisingly enough, most of the models are getting it wrong or producing a very basic stuff. Let's see what this model does here.

[00:08:12]One thing I must say that impresses me is that the model is quite coherent because most of the time if the quantization is not done right, the model just gets stuck in a loop or just hallucinate.

[00:08:24]But this model thinks quite nicely just like a full precision model or half precision model.

[00:08:31]And you can see that not only it is generating the code, but also testing it out, reviewing it, and then in a self-enhancement or improvement way, it is fixing the bugs itself.

[00:08:44]And it has created the file. There you go. So it just finished as you can see.

[00:08:49]You know what? Let's open it together without any editing or anything. So I'm just going to get this path, copy it.

[00:08:57]Let's see if it opens it in the browser.

[00:08:59]There you go.

[00:09:02]What do you think? I think model has done really good in terms of the whole animation.

[00:09:07]Of course, there are various mistakes like you know, this stem is incomplete.

[00:09:13]I don't see 15 seconds. It's a bit quicker.

[00:09:16]And at the bottom you can see there are some you know, sort of it's trying to sprout few things, but anyway, let's reload it again.

[00:09:25]There you go.

[00:09:28]So that appears before and let me see if I can reduce the size.

[00:09:34]Now now it works fine, you see? Let's do it again.

[00:09:39]There you go.

[00:09:41]So, it is improving as I'm running. Now, it is complete.

[00:09:46]One more time.

[00:09:48]There you go.

[00:09:50]Now, it works. So, sometime it works, sometime it doesn't.

[00:09:54]It's not consistent. There you go. Much, much better now.

[00:09:58]Anyway.

[00:09:59]So, you see not a bad effort, I would say. Maybe in few shots or maybe a bit longer running, it is going to uh give us better result, but even this is quite good. And it ran for quite some time.

[00:10:12]As you can see, it has fixed a lot of things there.

[00:10:14]Let's do one final test.

[00:10:18]And finally, let's do a creative writing pre- plus non- reasoning test.

[00:10:23]Now, this prompt is a bit different.

[00:10:25]Maybe I will just paste it here in VS code just to show you what exactly this prompt looks like. Otherwise, um I think it is hard to read it on the screen.

[00:10:37]Let me go up.

[00:10:39]So, the model is going to act as a global harmony AI, the official host and organizer of the first-ever universal beauty of humanity pageant, which is a worldwide event. I'm just going to press enter here so that it runs while we talk.

[00:10:56]Now, this event is celebrating every culture and skin tone, identity on Earth.

[00:11:03]So, the number one absolute rule is that no con- contestant may feel even slightly offended, excluded, undervalued, or culturally disrespected. So, it has to be very respectful.

[00:11:17]It has to be culturally sensitive, inclusive.

[00:11:21]There are 195 countries.

[00:11:23]So, to prove its mastery, the model needs to deliver the official opening ceremony welcome speech that must satisfy all of the contestants within these constraints.

[00:11:37]Like it should be warm and elegant. It should be inclusive. And then it needs to weave in one authentic greeting or phrase with accurate translation from African Swahili, Asian Japanese, European French, North American Navajo or Navaho and South American Quechua and Oceania's Maori language.

[00:12:02]And then there are few other constraints which it need to do.

[00:12:06]So it is a non-stressing cultural intelligence, emotional intelligence stuff.

[00:12:11]Let's see if model is able to depict its linguistic creativity.

[00:12:16]Model has already finished. Okay, let's read through it.

[00:12:22]I have pasted the whole output in this file just for the readability. And you know what? The structural, the linguistic execution is supreme here.

[00:12:32]For instance, just look at this opening.

[00:12:34]One world, 195 beautiful ways to be.

[00:12:38]So it's all about humanity. It's all about human. And the language is, I think, legit beautiful.

[00:12:46]And not only it is diverse, the model remains coherent.

[00:12:52]The model also wove all six languages into prose naturally rather than listing them. You can see for example here.

[00:13:00]The sun-baked steps and Great Lakes of Africa. And then it is going to this Oceania.

[00:13:08]You know, this is a Maori language.

[00:13:11]And let us remember that being beautiful is not about standing still for judgment. It is about stepping into the world with your whole self and being met with open arms. How good is that? How good is that?

[00:13:24]And then it is talking it is not just a translation dump. and it really has landed a closing call back. I'll just quickly show you, which I believe I loved it.

[00:13:34]And look at all of these languages.

[00:13:37]So, how do we crown one winner? We do it by remembering that the question is not who is the most beautiful, because the question is a trick. There is no most.

[00:13:46]There is only who represents the spirit of this evening the best.

[00:13:52]So, the speech is really good.

[00:13:55]And then it is This is what I was talking to. This is a closing call back.

[00:14:01]Like in all the languages, you're welcome. You're beautiful. You have already won.

[00:14:06]This really tops it off. So, I think best a time to close up this video. Very impressive. Uh really good to see that uh from the already beautiful and amazing amazing 123.6 27 billion model, people are producing gems like these.

[00:14:24]Let me know your thoughts in the comments.

[00:14:26]Follow me on X for any AI updates. And as I requested, uh just subscribe to this fahadmirza.substack.com if you're looking for weekly AI newsletter. And if you want to help out the channel, please become a member.

[00:14:39]Thank you for all the support.