Can A Frontier AI Model Reproduce a ML Research Paper (KIMI K3)

[00:00:02]I got early access to Kim K3 a few days ago before it went public.

[00:00:07]I've been running it for the last few days. And before I show you what I actually tested [music] it on, I want to walk through what this model is because the parameter count is uh very interesting to say the least. Here's the real question I want an answer to and it's bigger than just this one model.

[00:00:23]Can a frontier model actually reproduce a novel [music] machine learning paper from the mathematics alone with no reference code with nothing it's already seen a thousand times in training? Every launch this year gets tested the same way. Build a game, build a website, look impressive for 30 seconds. Almost nobody asks whether these things can take a paper [music] they've never seen implemented anywhere and get the math right on the first real attempt. K3 [music] is just the first model I'm running this question on. Kim K3 comes out of Moonshot AI, the team behind Kim K2, Kim 2.6, and 2.7, which were already very solid open weights models. The headline number on K3 is 2.8 [music] trillion total parameters, which makes it the largest openweight model released so far. But total parameter count [music] on its own doesn't really tell you much. The number that matters most is active parameters per token. Meaning how much of that model is actually going to do work on any [music] single answer.

[00:01:23]Kim 2.7 sits at 1 trillion total with [music] 32 billion active. GLM 5.2 is smaller overall at 753 billion total, but its active parameter count is a lot higher. And that's a big part of why GLM 5.2 punches above its total size. So with K3 at 2.8 8 trillion total. The question isn't whether it's big, it's whether Moonshot pushed the active parameter count up along [music] with it or just scale the total and kept the same parse ratio as before. If it's the second [music] one, K3 gets more knowledge packed in without necessarily getting smarter per token. The architecture rumor is that K3 is an ultra sparse mixture of experts model using a hybrid FMLA and KDA [music] style attention. Moonshot has already published research on this kind of linear retention approach. The point of that combination is to let the model [music] carry much bigger toll capacity without making every single response cost as much as a [music] fully dense model that size would.

[00:02:24]That architecture is also what's expected to give K3 a 1 million [music] token context window. The catch with any 1 million context model is routing. When you have that many experts and that much context, a bad routing [music] setup can make the model mildly inconsistent. It might nail one coding task and then pick up the wrong experts entirely on the next one and lose the thread. That's actually shown up before in Kimy's own agentic coding rankings on agent arena where Kimmy 2.7 codes [music] sits around rank 20 and Kimmy 2.6 around mark 24. Agent Arena isn't a static leaderboard either. It tracks real agent behavior over time using signals like downloads, disapproval, [music] retries, and how steerable the model is mid task.

[00:03:08]So, it's a decent proxy for whether a model can actually hold it together across long coding sessions, not just answer one prompt. Well, there is also a rumor going that K3 might be actually a distilled version of Fable 5. Nothing confirms that, and I'm not going to pretend it's settled, but [music] it's worth mentioning because it shapes expectations. Fable 5 is genuinely excellent in planning and reasoning.

[00:03:30]Arguably ahead of GPD 5.6 soul on that front, but it's not particularly strong at coding output on its own, which is why a lot of workflows pair Fable 5's planning with a stronger coding model.

[00:03:42]If K3 inherited any of that reasoning strength while also being tuned specifically for coding and dual use, that would be the best case [music] scenario. The more grounded comparison though is against GLM 5.2 two max since that's [music] the open model K3 actually has to be the matter in this tier. If Moonshot gets the routing right in the active parameter count scales with the total K3 would easily fit in the top 10 on agent arena possibly closer to GM 5.2 than the closest Frontier models. If it undersshoots [music] on routing, you'll see the same pattern Kimmy has had before. Flash is [music] a very capable model that falls apart on longer sessions. That's the context. And now here's what I actually did with my axis. Like I said, most launch videos this week are going to show you K3 building a game or a 3D scene. And that's a fine way to see the vibe of the model, but it doesn't tell you much about whether it can reason through something with a right answered at the end. So I wanted to test where I already know what correct looks before I start and where the question being answered isn't just about K3. It's the question I asked at the start. Can a frontier model take a novel paper and rebuild the math correctly with nothing to copy from? Before I get into the test, let me explain what this paper is actually about. Because the terms KDA and MLA are going to come up a lot, and I don't want to throw guardian at you.

[00:05:04]Every time a model like this generates text, [music] it has to look back at everything it's already written or read to figure out what comes next. That's what attention does. The normal way of doing this called full attention means the model keeps a running record of every single word it has seen so far.

[00:05:20]And every time twice a new word, it [music] checks that new word against every single one of these old words. If you written a thousand words, checking the next word costs you a thousand comparisons.

[00:05:32]If you written a million words, well, it cost you a million comparisons. And that record that's keeping in memory grows the same way. That's why models slow down and eat more memory the longer a conversation or document gets. The cost grows much faster than the text does.

[00:05:48]Kimmy delta attention or [music] KDA is Moonshot's attempt at avoiding that problem instead of keeping a full record of [music] everything and checking against all of it every time. KDA compresses what it has read so far into a small running summary and it updates that summary a chunk at a time as it goes using a gating mechanism which is just a set of rules for deciding what's worth keeping in [music] the summary and what can be dropped. That's why the style of attention barely gets more expensive [music] as the text gets longer since it's not comparing against everything. The trade-off is that a pure summary based approach like this can lose some fine details. So of course Moonshot didn't go all the way in. Kim linear uses a 3:1 mix, three layers of the cheap summary based KDA for every one layer of the expensive but more precise full attention. The full attention layers act like periodic check-ins that catch everything the summary might have smoothed over while the KDA layers do most of the heavy lifting capabilities cheaply. That's the whole idea. cut the cost of remembering long text by mostly working off a compressed summary instead of the full history and sprinkling just enough full attention to keep it accurate.

[00:06:57]Moonshot's own numbers claim up to 75% less memory spent on that running record and up [music] to six time faster generation at the million token range.

[00:07:06]This paper works as a test case because it's recent enough, math heavy enough, and specific enough in its claims that there's no faking a correct answer. I gave K3 this paper and nothing else. No reference code, no repo like I said and no browsing access. So here's what K3 actually has to do. First implement the KDA layer from the papers math alone, the chunked recurrence update and the gating, nothing else. Then I check it the same way you check any kernel implementation. run identical random inputs through K3's version and through Moonshot's own open-source reference kernel and see whether the outputs match within a small numerical tolerance on both the forward pass and the gradients.

[00:07:51]Getting a gated linear attention mechanism with chunked recurrence to match a reference exactly is a genuinely hard problem on its own with no training required. Once it's numerically correct, I test the actual efficiency claim the same way the paper does. As a property of the mechanism itself using random unconstrained weights, I benchmark T3's implementation against a plain full attention implementation at a few increasing sequence lengths and measure memory used and time per decoded token.

[00:08:20]Full attention should blow up quadratically. GDA is supposed to stay [music] close to flat. That shape is what I'm checking for, not the paper's exact 75% or 6x speed.

[00:08:32]Since these come from a scale I can't touch, no training loop anywhere in this [music] test. This is the part where I actually care about answering. Not just can K3 use KDA, but can it read cold math? It's never seen implemented anywhere and gets it right. So here are the rules. No reference implementation.

[00:08:49]Even if the authors published one, K3 only sees the paper. No browsing access during the task. and it [music] has to implement the KDA layer from the math alone. No training involved anywhere. I check correctness by diffinging its output against Moonshot's own open source reference kernel on random inputs. Once it's numerically correct, I benchmark memory and decode speed against a plain full attention [music] implementation using untrained weights and I'm timing everything in log in every point where I had to step in.

[00:09:19]Kimmy thought for 40 minutes while reading the paper and then verification came back first. [music] It made six test cases checking the chunked implementation against the raw recurrence in float 64. Different sequence lengths ragged sequences with and without [music] a starting state. It failed multiple times but at the end also expressed the relative error around 10us14 to 10us15 on the output the final state and the gradients which surprisingly pushed the precision limit of float 64 itself. The chunked math D3 wrote is an [music] exact match to equation one in the paper. The benchmark compared KDA against plain full attention. Same model size from 1,024 tokens up to 32,000. Full attention's decode time per token grew almost 16x over that range. Its cache grew 30x which is what quadratic scaling looks like in practice. KD's decode time per token moved about 2%. Its state size stayed at.5 megabytes the entire way at 1,000 tokens and at 32,000 tokens.

[00:10:22]The flat line is the paper central claim a fixed size running summary instead of a growing history. The absolute numbers here won't match the papers since this is an optimized Python on a single consumer GPU. But the shape of the two curves is what I was checking and it lines up. So back to the question from the start. Given only the paper, no reference code, K3 got the math right on the first attempt. It matched the reference kernel to near machine precision and produced its scaling behavior in the same direction as the paper. and it used 30% of my 5h hour limit on the max plan. And that's K3 beta tested a day before public release.

[00:11:01]I'll put the reproduction code and the fullr run logs in the description so you can implement it yourself. So let me guys know in the comments if you want me to make a series about this test.

[00:11:11]Bye-bye.