The Desktop Frontier — Ahmad Osman, Osmantic

[00:00:01][music] Hey everyone, we are about to start this presentation.

[00:00:17]Uh it's called the desktop frontier [snorts] and um [clears throat] it's basically about where we started and how far we've come with local and open source models. Um how many like just a quick question how many of you here follow me on X?

[00:00:42]>> I'm I'm amazing. Love you all. Love you all. Uh so you know I sometimes every now and then I would say a prediction uh here is a new one within roughly 18 months we are going to have the equivalent of GLM 5.2 class intelligence running on a single RTX 5090 with 32 GB of VRAM. Um that's basically late 2027. Um this this is conservative. We might actually get there faster.

[00:01:17]So uh you know for a long time uh the story has been bigger models bigger models bigger models how can we get to the next 5 trillion how can we get to the 20 trillion and I'm not saying that there won't ever be like a gap between frontier intelligence um and u you know open source models there will always be a gap but that gap um will shrink and the efficiency of the models will get exponentially better.

[00:01:51]So the the term that I like to think about is impact per parameter. Um you know what capability are we talking about? What could a model do? Um what footprint like hardware footprint did it have last year in comparison to now? and uh what hardware does that use and what hardware does it need to use a year ago and uh you know are we moving down for the same kind of quality on the on that hardware again as I was saying earlier I used to run lama 2 on an RTX 3090 it's now running qu 3.5 3.6 6 27 billion parameter. That's better than Lamas 405.

[00:02:35]That's a 400 billion 400 billion plus parameters model that you beat with a 27 billion parameter model a year and a half after.

[00:02:48]So um yeah, as I was saying, similar capabilities are moving into smaller hardware footprint. Um benchmark scores are one thing but also you know a year ago this time a year ago we didn't have any local models that were able to successfully run within clo code right it wasn't until GLM 4.5 that came out in late July and GLM 4.5 air required at least four RTX uh 3090s or an RTX Pro 6000. Now that footprint for hardware is not needed anymore. All that you need is a single RTX 1390 1590 and you have something much more capable, much more intelligent. So is this trend just random or is there more to it?

[00:03:41]That's a question that everyone should ask.

[00:03:44]Um, is it just by by random chance that we've gotten this far from models that were weren't able to sustain more than 4,000 um tokens in terms of context lengths and now we have things that are million parame million uh tokens locally on your hardware that you own. It's not by chance, you know, it's not just a coincidence that we got here. Um, there is research being done. There is efficiency gains to be made. There are architecture hacks that compound and they will continue to compound.

[00:04:21]And I think I like this line. It's not that small models are beating big models. It's that newer, more efficient models are beating older, less efficient ones.

[00:04:33]Uh so yeah capability density uh is you know the literature I back this up with uh uh nature machine intelligence uh calls this pattern densing law and uh basically um you know every three and a half months we are having 50% fear parameters whether that's in dense or activated that's a different story but we're getting way more intelligence out of the models that we're running.

[00:05:09]So, you know, right now where we're at, it's uh GLM 5.2. That's uh our, you know, biggest player and um it's 744 billion parameters total with only 40 billion parameter activated. And that supports up to 1 million contexts. You can run this in MVFP4 on a machine uh on a GGX station or on a server with eight RTX Pro 6000. That's something that you like a GX station is something that you can sit under your desk and it's running this kind of frontier intelligence.

[00:05:43]Whether you know it it's on one benchmark it actually beats GBT 5.5 extra high. Doesn't that mean that we're getting somewhere with local and open source models that we can compete with the frontier that we're not that far off from the best that you can get from the cloud?

[00:06:02]We also have Neatron 3 ultra which proved that NVFB4 training more efficient training can be done on uh on hardware right that's that's very important that means that the footprint even for training these models for fine-tuning them for making small and specialized models as I was talking earlier could be more efficient could be done cheaper and could be you know could deliver you value in terms of economics way sooner or you know for much less money than you used Yeah. So, you know, again, Lama 2, uh, that was a 70 billion parameter model.

[00:06:38]If you try to run that right now, you're you'd laugh at it, right? That used to take eight RTX 1390s to load up and it those same eight RTX3090s could run something like 15 parallel agents right now with Quen 3.5 27. That's that's a massive jump in terms of performance gains. Um so the densing law basically means that we have similar or better um capabilities with significantly fewer parameters. That's the impact per parameter. As I was saying I want everybody to live here thinking about this term and you know thinking where are we going to get a year from today as I was saying earlier everyone here has a phone. I'm assuming raise your hand if you have a phone.

[00:07:25]If you didn't raise your hand, we know you lie about other things as well. So, come on, guys.

[00:07:31]So, [clears throat] you know, you can run you can now run GBT40 quality on your iPhone. That that's massive. That thing require data centers to serve. So, why wouldn't you invest, you know, in sovereign [snorts] AI? Why wouldn't you as a consumer, as an individual, as a smalls size business, middlesiz business, enterprise, why wouldn't you want to be in control of the models that you're on?

[00:07:59]Why wouldn't you want to make sure that nothing gets taken away from you? That every little thing can be optimized for you later on, that the performance gains can be made specially and specifically for your use cases, and that you can save more money that way in the long run.

[00:08:19]and you know ODS for consumers it's basically the way that we support individuals but enterprises also and I think that there is something that we like as a community we need to think about deeply we need enterprises for open source AI to win we need these people that are using the cloud right now that are basically supporting data centers being built for cloud providers to come on this side to own their own hardware to own the stack fully end to end so that we can keep delivering open source models so that there is an incentive for open source providers to actually come up with models so that we can come up with new licenses that allows open source to thrive and so again um open weight and the frontier I think I yeah sorry that was a misclick um [clears throat] you know so smaller models started bunching above the weight after lama 2 with mistral 7B one of my favorite models if you try to build that model right now and um clo code or oven code it's not going to work but it used to take so much in terms of hardware right that you would now get from a 9b mill model that I can run with telegram with or with hermes for example and uh do a lot of stuff with so we've come a long Okay. Uh we had that we had mixed trial 8 by 7B which you know everybody knows is an MOE. Then the progression went uh from that to Lamas 3 you know Lamas 3 8B was one of my favorites still is um it had unique identity in my opinion. Uh then we had like the 70 billion which was like the the thing that I would run basically on my 8 RTX3090s at home. Then there was like the 405 the 400 billion plus parameter lamas which again required a lot of hardware and if you put it now against 3.5 the 27 billion parameter would lose against it that's in the span of what two years two years and some no I think I think less than two years that's summer 2024 to March 2026 that's uh that's about 21 months and the next big thing in my opinion gamma 27V and then we had the Quinn 2.5 and that that was the moment that I was like okay we actually are making progress and the gap was shrinking between open source models and uh the frontier uh really lamas 3 saved like you know it really helped us a lot and then um Gwen 2.5 delivered a massive improvement and there was a lot of fine-tuning and experiments that could be done on that one there was amazing papers and um they helped the community immensely in my opinion.

[00:11:12]Then the next big thing was Deepseek R1 in my opinion and uh reasoning becoming something that you can run at home. That was a massive MOE almost 700 billion parameters. Um you know you had to have like a very beefy server to actually get it up and running. Um and then you know the improvements that came from just more training on that one and Deepseek R1 that was released in May last year made massive jump again. So it showed that post training could deliver more improvements on the same on the same checkpoints.

[00:11:53]Then GBT open source like GBT OSS 12B.

[00:11:57]Anyone remembers that one from last summer? Yeah.

[00:12:01]Nobody here used it. Come on, guys. I I need some help here. [laughter] Uh it was it was it was one of the first open source models that were able to successfully do tool calling. Um and uh it was a step forward. It showed us that we can do more with uh with the hardware that we have at running at home. Uh that was a footprint shift right from like you know that massive 700 billion parameters deepseek uh R1 that was yeah 671 billion parameters to something that was 1/5 of its size in GBTSS with comparable maybe better more agentic performance. Um [snorts] then the moment of uh quinc 3.5 that's 397 that's three that's 397 billion parameters uh that's uh that's a BVOE and uh you know what's funny is that about uh it's about 15 times the size of uh the Quinn 3.6 and I'm here I'm comparing 3.5 to 3.6 6 of the dense 27 billion parameter model and that dense model beats it and that dense model has 40% higher number of activated parameters. So it's not that far off.

[00:13:24]That's massive amount of performance gains and a very small amount of time with massively different footprint in terms of hardware uh requirements. And that trend happened in like what two three months. So you know how far could we go from here? Um how far before we get to you know uh a recent model that there was some news about you know that is uh finally relaunch the game. How far before open source delivers something of that quality that you could run on your own hardware and you can control and will not be taken away from you and will not refuse a request from you.

[00:14:08]So again these are just some benchmarks where you can see that an iteration on the 27 billion parameter model a little bit more post training proved it across all benchmarks and made it one against a model that is almost 15 size for 15 times its size.

[00:14:32]And again, remember this is 27 billion parameters activated versus 17 billion parameter activated. It's still massively the same amount like you know it's it's only 40% less in terms of the amount of time it would take to process things but it's 15 times smaller. That's a lot.

[00:14:53]So again um how long until the prediction I made earlier becomes plausible when I said that we're going to have the equivalent of GLM 5.2 2 running on an RTX uh 1590.

[00:15:06]This is the mass 17 months and this is a conservative math. Uh earlier this year in December, I had a a very viral post that I predicted that we're going to have the quality of OBS 4.5 running locally at home on a single RTX uh Pro 6000. That happened by March.

[00:15:30]So a question um hardware purchase today does it get more valuable as models become more efficient and smaller in size?

[00:15:43]That's a good question. So why are you funding other people to build data centers so that you can subscribe to them and pay subsidized tokens and then later on get those subsidies are going to go away and you're not going to be able to run those models and they will have so many limitations. So might as well ask yourself why not own the hardware yourself and be in control.

[00:16:07]Um, so yeah, the forwardl lookinging question is basically what will a DGX station be able to run in three, six, 12, 18 months from now? That's something that there is a reason that I'm not selling any of my RTX3090s if you follow me. And I have a lot of hardware, guys.

[00:16:22]U but I'm interested in seeing what I could do with them in a year or two from now more than in the amount of money I would get for them today.

[00:16:31]This is not a financial advice by the way. Like let me make that very clear.

[00:16:36]Um so yeah um the disk side frontier potential um you know an Nvidia GX station could run a lot of uh today it could run GLM 5.2 to what will it be able to run tomorrow 6 months 18 months two years from today we know that you know RTX3090 [snorts] is the amber uh architecture from 2020 sells at higher value than MSRP today and it's still being utilized for a lot of use cases so what well at GGX station the actively developed blackwell architecture will be able to run in a few months a couple of years that's a Good question.

[00:17:18]So the question you have to ask yourself um if an RTX30590 with 32 GB of VRAM runs in the equivalent of a GLM 5.2 and 18 months and this is the question that everybody should be asking themselves and I want you all to be looking at the screen taking this very seriously. Okay.

[00:17:40]Should you buy a GPU?

[00:17:44]>> Thank you.