The AI Model Wasn’t the Breakthrough...

[00:00:00]Something really interesting just happened, and not many people are talking about it. A team called Impossible Research just created an AI harness that achieved like 99% on the ARC AGI 3 challenge, which if you know anything about that, it's kind of So, let's unpack all three of those things. What the ARC challenge is, what harnesses are all about, and why I think this is such a huge deal and maybe the biggest untapped opportunity in AI right now. So, if you've never heard of it before, this is ARC. It's an AI benchmark designed to kind of test more general intelligence, not how much the AI already knows, but how well it can adapt to a novel problem and figure out something unfamiliar, which is what humans do all the time, but AI has historically been pretty bad at it. So, ARC AGI 1 launched in 2019, and to us it seems like a fairly trivial and simple problem. You get one example where you see the pink blocks, and the solution is over here with the same pink blocks and the yellow ones, and then you get another example, and then it asks us to solve the final one, which is pretty easy to see that anywhere there's three pink blocks, you fill in the square with the yellow one. The AIs traditionally had a very difficult time with this type of a problem, and it took them several years. So, if we look at the leaderboard, now it has been completely saturated, but the older models had a difficult time. So, what we're seeing here is the percentage on the vertical axis that the models got right, and on the horizontal axis, how much it cost to get that result. So, you can see older models like GPT 4.5, they maxed out at like 10%, and it cost something like 30 cents, whereas the latest models like GPT 5.6 solo on extra high get 97.5% at about the same cost. So, it's basically a saturated benchmark. So, last year, ARC introduced the AGI 2 challenge. These were far more complex problems. You can see there's more difficult reasoning in terms of which color to give to the shapes depending on how many distinct holes they have in the middle. And again, the humans or the models are not given any instructions, so they don't know what the shape of the solution even looks like. They just have to use their intuition to figure it out.

[00:02:19]And still, for humans, we can essentially get 100% on these problems.

[00:02:23]It's not that difficult. But for the models, it's a big challenge. And if we look again at the leaderboards, the Arc AGI 2 has just recently, over the last maybe 6 months, started to be quite saturated. So, the models that were released this year, like GPT-5.6 again, can score about 92%, which is almost as good as the humans at an average cost of like 1 and 1/2 dollars per task. But again, going back to GPT-4.5, it couldn't even score 1%. It spent $2. So, you can see how the models have become much better at this over time. So, of course, Arc introduced the AGI 3 challenge just a couple months ago.

[00:02:59]These are far more complex problems. As they say, instead of solving static puzzles, agents must learn from experience inside each environment, perceiving what matters, selecting actions, and adapting their strategy without relying on any natural language instructions. These require interactive solves. They don't tell you what the solution is. You have to use the arrow keys.

[00:03:20]And in this one, you have to adjust the shape in the bottom corner to match the shape on the screen. And once it's done, then go to the solution.

[00:03:30]They're far more complex problems. And as a result, even the state-of-the-art models as of July 2026 are still only scoring about 8%, and they're costing $25,000 in token costs to get that 8%. And it's worth noting here that for AGI 3, there are 25 public examples that users or humans can play. Each one has a different type of solution, it's a different type of game, and again, they don't tell you what the solution is.

[00:04:02]There's also a lot of semi-private tests that are not available to the public.

[00:04:06]So, when you see that GPT Soul solved about 8%, that's including the private unreleased examples. It scored about 13% on the public examples. Well, fast forward to this week, there's never a dull moment in the AI world, and the team at Impossible Research announced that they achieved 98.9% on Arc-AGI 3, with a bunch of caveats that we'll get to in a minute. But, the way they did this is kind of a brilliant loophole. It wasn't by training a better model, which is what the test was designed to measure. They used off-the-shelf models like Fable and Soul, but they created a much more powerful harness that made these models better at this particular task. Harnesses are the future of AI, in my opinion. Over the next 6 or 12 months, I think we're all going to start hearing a lot more about harnesses, because they are a gold mine that allows us to unlock so much potential in the existing language models. A harness is really just the software that wraps around an AI model that guides how it works. You can see the definition here from an Anthropic article. An agent harness is the system that enables a model to act as an agent. It processes inputs, orchestrates tool calls, and returns results. So, the model, whether it's GPT or Claude, it supplies the raw intelligence and reasoning ability. The harness is what decides what information the model sees, what custom instructions it receives, what tools it can use, like all the memory that it can retain, how it ultimately checks its work, and what happens once it produces the output. And it's all designed for a specific task.

[00:05:43]And then the agents, we all hear a a about agents. Agents are really just the combination of the model and the harness working together for a particular goal.

[00:05:52]So, Claude is a model. It can do all kinds of things. Claude code is something that you install on your computer, and that's the harness for the model. So, the model can write all the code, but the harness gives it that repository on your computer. It gives it the command line interface or the terminal. It gives it access to your file system, so it can create and edit files. It adds lots of instructions and predefined tools and loops that reduce the errors and achieve better results at coding. So, Schema's core idea was kind of brilliant. Instead of asking the model to keep all of this game knowledge inside of its own context, which was the spirit of the challenge, instead they built this awesome harness around it that, as far as I know, it does not hardcode the rules of the game and say how to beat each one, which would be cheating. Instead, they just gave it this harness, which has a broad set of tools that allows it to write code and to essentially spin up a simulated version of the game to reason about how you might solve different puzzles. And then, only once it's really confident that it knows how to solve a particular game, does it actually execute that plan in the real environment. Haven Fang on Twitter, one of the creators of this, when he posted about it, described Schema saying it can handle the state and mechanism in one editable program, a symbolic world model. It designs experiments to verify hypotheses, back test the program against history, and plans inside its world at zero action cost. So, the harness allows it to do all of this on its own before it ever enters the real game, and that's where the power comes from. They also put up a website or a paper that goes goes into great detail about how Schema works. And what seems most impressive to me about what they've done is they show the results versus humans on all 25 of the games. So, for each of these charts, the pink line is what Schema did and the gray line is what humans did. And it's measuring the number of actions it took before it conquered the level or achieved the challenge. And in almost every case, after being able to plan and to think about the problems, Schema with frontier models like Opus, Fable, and in some cases GPT-5.6, was able to solve these puzzles in less actions, like less keyboard movements, than humans were able to do. And keep in mind what we saw earlier that GPT-5.6 by itself, without this harness, was only able to achieve 8% or so on the full benchmark and 13% on this public benchmark which Schema tested on. Now, there's a lot of caveats to this. This was a self-reported result, meaning it didn't go through the official ARC testing process. It was only tested on those 25 public data sets, not on the full semi-private data set.

[00:08:38]And I think that the president of the ARC challenge on Twitter has a great response to this.

[00:08:42]The TLDR is that without inspecting the code or harness, it's tough to tell how much human intelligence was baked into this. There are signs it's non-zero.

[00:08:51]But regardless, it is really interesting work. I think it's worth like pausing and reading some of his responses to the paper that they presented. But I think what we're seeing in this result is the power of the harness. How you use the model is at least as important as the raw intelligence of the model itself these days. And I think we're really only beginning to scratch the surface of what's possible here. I personally think harnesses are the next frontier. This is what will allow us to unlock so much more value out of the existing models, possibly even taking them to the next level. I I don't personally think that LLMs will get us all the way to AGI or ASI, but maybe with the right harnesses, we can get much, much closer than we can with the raw models. At the very least, I think harnesses will replace software as the primary way that humans interact with computers. Like all that deterministic, general-purpose software is probably going to die in an era where anyone can write a custom application essentially for free. I've talked about this a lot in the past as someone who used to write this software. But maybe what takes the place of SAS is going to be HAS. Like what if we have harnesses that use AI to just do whatever needs to be done within a particular domain or workspace?

[00:10:08]And so far, that has just been coding.

[00:10:11]Like we've all got to break out of the idea that harnesses are for coding because there's so much more potential out there. For example, how cool would it be to have like an educational tutor harness that can teach you anything? It does research and builds a curriculum.

[00:10:27]It can give you interactive quizzes that meet you at your level and track your progress. It has a memory about your learning style and what you're struggling with. That would be awesome.

[00:10:37]I would love to have that. Or maybe a video editing harness. Like I think the future of AI video editing is not just to add AI tools into Final Cut. It might be an entirely new, reimagined harness where you just drag your clips into it, speak a couple sentences about what you want to have as a final product, and the harness can just run tools to make cuts and find B-roll images and generate titles and review the final product until it's done. That would be amazing.

[00:11:06]Or a bookkeeping harness or a health and fitness coach would be awesome. And there's just like infinite ideas on what we could be doing. And the capabilities to make this already exist in the model right now. We just have to start thinking more creatively about how to guide them towards these outcomes that we'd like to have. And that's really exciting. So in the past, we had software engineers. And then more recently, we've had prompt engineers.

[00:11:30]And I think we're about to have harness engineers. And I'm ready.