Next.js AI-Ranking: Kimi K3 + OpenCode Is BETTER Than Claude Code Fable / Codex?

[00:00:00]Hey everyone, there's a bunch of big updates here in the world of AI as it relates to well specifically Nex.js, but more broadly web development or you know building full stack apps. So in this video and the next few videos as well, I want to go over some of these updates and how we can optimize our text stack our our setup essentially to get the most out of these AI updates. So you may have seen that there is a new model here Kimmy K3 and surprisingly it's performing very well. it on some of the benchmarks it performs better than the other Frontier models like Fable which already had a lot of hype but it's supposed to be even cheaper and it's going to come with open weights. So this is actually really interesting and specifically for NexJS it is currently the best performing model on these NextJS uh evals. We'll talk about that in a second. It's basically uh well a leaderboard of the best performing models on some particular tasks that the NexJS team publishes here on their website. But not only that, uh there is also this arena.ai uh leaderboard that actually also has Kimmy now as the number one in the front end code arena surpassing Fable 5. Does that mean that you have to use Kimmy or even Fable for your NextJS project?

[00:01:10]Right? Because you could say, well, yeah, I want the best performance, so I should be using the the models that are highest ranking on these leaderboard. So here the forell CEO published about it and uh he mentioned that this is the first time that an open model is ahead of all proprietary ones for this comprehensive web engineering benchmark.

[00:01:30]Now he does add here that benchmarks don't always tell the full story and we'll talk about some of those things but it could still be a very good indicator. So let's actually talk about uh these evals here. Uh let's start off with this Nex.js uh specific one. But my guess is if it performs well on NextJS, I think it's likely that it performs well on the other frameworks as well. Uh but here for NextJS specifically, you can find it on their website. They have a bunch of evals here. So these are performance results of AI coding agents on Next.js code generation and migration tasks measuring success rate and also execution time. Okay, so what does that mean? Well, if we scroll down a little bit, we can see for these agents and well actually the models and then in which agent it is being used like the agent hardness, we can see if we pop open this uh this one for Kimmy K3, we can see the tasks that they're being evaluated on. So these models, they're getting these tasks for example avoiding uh fetch in use effect, right? So typically we do not want to do this in an X.js app. But in the training data for a lot of these models uh because they're trained on you know typical React code and often like V code in which you have more like client side data fetching they may see a lot of fetching in use effect and they may tend to carry that over into next.js app, right? But actually we often do not want to do that in nextJS. So that's a task that they're being evaluated on and if they get it right they get a score of one. So you can see Kim K3 did it passed that test and it passed these other ones as well like preferring a server action and uh avoiding get server side props right this is a bit of an old school one uh preferring the next link and let's see and um also some new ones right so we have the use cache directive now it's pretty new the training data is probably from before use cache was really introduced pretty impressive that it got that right there is also a hard app router migration task so it got almost all of this Right? So you can see there's a one here for almost all of them, including things like optimizing the partial pre-rendering shell. Right?

[00:03:30]Basically, we want to have as much of the app part of the static shell. Uh that can be rendered during build time.

[00:03:37]We can put it on CDN. It's much more efficient than having things be dynamically rendered. So there's optimization task for that and it's able to do that as well even for a few transitions. So the only two tasks that it was not able to do properly was the proxy middleware, right? So in the latest Nex versions, it's not called middleware.ts anymore. It's proxy.ts.

[00:03:57]Basically just a naming change. But yeah, if it's trained on older data, it may get it wrong and looks like it got it wrong. And also there is something with instant navigations now. Um, so that is also something that it was not able to do, but it's because it's really new probably. Uh but overall it had a pretty high su success rate here of 92%.

[00:04:16]And if we take a look at Fable here, so here below there, number two on this leaderboard is Cloud Fable. And we can see that it also almost gets all of that right. It has a bunch of ones here for all of them except for well here it actually gets a different thing wrong.

[00:04:32]The few transitions for some reason Cloud Fable is actually able to do the what is it? The um proxy middleware. It actually got it right. I don't know why uh is able to get it right and Kimmy isn't, but um it got something else wrong. So, they actually have the same success rate. They both have two wrong.

[00:04:48]However, Kimmy is a bit faster. So, I guess that's why it's on top here. My only guess would be why cursor is actually not at the top because it has the same success rate here. And it seems to be even faster than Kimmy K3. So, I'm not sure if there is some weighing involved here of these different tasks.

[00:05:04]Uh maybe it's because it was not able to do the instant here and it was not applicable. Maybe it just wasn't able to do the task, but actually cursor is seems to be performing well on this leaderboard as well. Our codeex here with a GPT 5.6 Soul Ultra is number four, but it has the same success rate.

[00:05:20]It just take seems to take a little bit longer. I think that's just why it's number four here. Now, we see something else here which is important. You're going to see this every time you're going to create a new NextJS app, which is the success rate with agents. ND. So, I want to talk a little bit more about where this is going to change. This is going to affect how you set things up when you're building new project. We're going to see that for nextj specifically, but we're also going to see that for third parties like superbase and my guess will be other ones as well, which is basically that um documentation will be included will be bundled with the app. So if we open up Kimmy K3 again, if we scroll down a little bit, so it got the proxy middleware task wrong, but if the next sdocs are bundled with the app, it actually gets it right. What I mean by that is um I have if I start a new nextjs app, right? Let's quickly do uh mpx create next app, right? I'm just going to use latest one. I'll put everything in this directory here. I'm going to just use the recommended defaults. And now we have our next s app. So a lot of it looks similar as uh what you're used to, but we do see a few new things here in the latest versions of next.js, which are actually these uh agents related files, markdown files. So if we take a look at agents.mmd, if you're using an AI coding agent like uh codeex or open code or other ones, they should be able to pick up on the fact there's an agents.mmarkdown uh file here in this project folder. And so when you start chatting with your coding agent, it should be reading this. The exception here is clot code. Clot code works with a clot.md file. So clot code will scan the repository here for clot.md file and it will find it here. And what are we doing in here? We're just pointing to the agents.m MD file. So all the coding agents ultimately will end up with this file here. What is being set in here?

[00:07:06]Well, we are telling the coding agent this is not the next.js, you know, because from the training data, this could be like 6 months ago or even a year ago. They've seen a lot of next.js related code that is simply outdated.

[00:07:18]It's still, you know, there's a bunch of updates into Next.js jazz every well few months or so and it's simply not going to match exactly what they've seen when they were trained. So what are we doing here? We're telling the AI coding agent, hey, if you want the right information about this specific nextjs version, read the relevant guide in note modules next docs before writing any code. So basically the next s documentation right here if I go to uh docs right. So here you can see it's based on the version right. So here is is the version that I'm using as recording. The documentation here is now included here in the app itself. So if we go to note modules and then we have docs here. So this is where the agents are being pointed to and here we see an index.md file and this is an explainer about the nextjs documentation. So here what is nextjs? The agents are going to read this how to use the docs. There's a difference between app router and pages router. So here they are then further being instructed to look up the other files in here. For example, uh here for the app router, there are a bunch of other files, right? And here we have a lot of information about uh server and client components, for example, how to fetch data in here, right? In server components and here we also have proxy.md, right? So here it says something about uh starting with next.j 16 middleware is now called proxy. This is what that means here if we take a look here. So here they have this asterisk here. So if the project includes agents.md, Kimmy does get it right. Right? That's why it's green to highlight, hey, there's a difference here between including agents.mmd and not including it. By by having this included, it actually got it right. So that's important because that is really that's a more realistic setup, right? So here we can see success rate with agents.md for Kimmy was 96%. So the success rate without it was a little bit lower. If agents.mmd is included, it's a little bit higher. The same is true for fable 5 here. Um if we scroll down, you can see few transitions. that gets it right if agents.md is included. So that is ultimately what matters I think when you're evaluating which model or agent you're going to use because typically you are going to include agents.mmd in the latest next s versions. So this is the real metric that matters in my view and you can see that they are actually all of these models and agents are actually all pretty good. a lot of these mid-tier models. I guess you know they're getting about 80% 75% but when agents ND is included it shoots up to well about the same as the front tier models in fact if we scroll down some of these lower tier models like cloud 4.6 which is a bit of an older model. It gets a lot of things wrong in Nex.js. So it's a lot of these tasks almost half gets it wrong. But if it comes with the nextjs docs bundled in there, we can see that it gets the same success rate as the frontier models. So what this shows in my view is the importance of having documentation related information bundled with the project. The agent is then able to increase its success rate significantly. So that's why I think you should take this hype with a grain of salt. Yes, Kim K3 and and Cloud Fable 5 are really good. But in practice when you're doing NextJS as long as you have the docks, the difference with the uh older models should not be as much. So I personally feel fine just using codecs or cloud code with you know a non top two model here cuz I think the success rate is going to be similar. 16.2 they started bundling the next documentation into your project. Now there are a bunch of other things actually here. Here, if we look at the updates here in the next JS uh dogs, there are a bunch of other things as well, which we'll talk more about in the next videos to really completely uh set up the text tag here uh for AI. They have a code mod here in case you're interested. But I think this shows that we're going to have more and more essential information bundled with the actual project. Actually, another good example here is Superbase. They have just released agent skills which is basically a bunch of instructions for the coding agents on when to look up uh the right uh documentation up toate documentation but they are also including some of the most critical information uh with the actual skill itself. So the agent does not have to go out there on the internet make a fetch call or something like that to get the latest docs. It's already bundled with the project essentially with the skill.

[00:11:28]So I'm actually partnering with Superbase on a few videos here to show these agent skills. So these are an open-source set of instructions that teach AI coding agents how to build on superbase correctly. And the reason this is so important is because uh these tasks that we see here that these are being evaluated on these are just you know very isolated specific nextjs tasks. But in the real world the complexity starts when you're integrating some kind of tool into the next app right building a full stack app with a database for example. So the agent needs to know how to integrate it and just knowing about NexJS itself is not sufficient. It needs to know about the specific tools it that you're integrating as well. That's so very important. A lot of you are using superbase. We do not want to leak user data. So security is really paramount.

[00:12:14]So here uh superbase agent skills an open source set of instructions that teach AI coding agents how to build on superbase correctly. So if you're using cloud code, codeex, copilot cursor or any of the agents that support this agent skills open standard, right? So a skill is just basically some additional context for the AI coding agent. Um it can be a lot of like a lot of uh context and the benefit essentially of a skill is that the AI coding agent only loads it when it thinks it needs it. So it's basically uh sort of dynamically loading more uh information into the contact. So if you have a query about uh RLS policies, it may be able to uh load more context about that on demand, right? So it's not unnecessarily polluting the context with a bunch of unrelated information, right? So in practice, I know a lot of you are struggling with this. You have RLS, right? So you have these rowle security in superb basically security related concept and it is a little bit tricky. Maybe you've tried handing that off to your AI coding agent, but even the coding agents sometimes made little mistakes, right?

[00:13:15]So they saw that agents skipped RLS policies on exposed schemas. They may hallucinate CLI commands that don't exist like superbase DB execute. They may create views with security invoker is true which actually silently bypasses rowle security. And sometimes they also just ignore the docs entirely. Maybe you've seen this where the agent thinks it can skip looking up information because maybe it already has information as part of its training data. Basically as part of well what it already knows about. So it needs a little push essentially to actually do indeed look up the latest information. Now they already had best practices uh for Postgress. So if you were using Postgress database, they actually already had a nice set of instructions here for your coding agent to help you optimize that. And that is still part of this actually. But this goes even further. It covers all of the superbase product and it encodes the security requirements in line. This is actually sort of bundling information with the project I would argue and teaches agents to check the docs before implementing anything. So what the skill teaches so if we install the skill everything critical lives directly in that skill.md. It's about 100 lines and they put it there on purpose. So basically bundling it inside of that. So these are not pulled from the docs on demand.

[00:14:29]They're loaded with the skill so the agent has no excuse to miss them. So here are some of them. So for example here in nextjs uh we we don't want to expose the servers ro key on the front end. It should not be using this next public prefix for environment variables cuz then it's exposed in the browser. So we moved everything critical into a skill.md itself where the agent reads it the moment the skill loads. So I think this is very similar as to uh bundling the docs uh with next. And here they have some benchmarks on how much it improved the correctness. I'll link to this in the description if you want to read it by yourself. But basically, if you were just using the AI coding agent without MCP, it gets about I would say 50 60% right. If you're only using the MCP, you can see it's only a marginal improvement uh here. However, if you have the MCP and the skill, it's a significant improvement across the board. Now, one other note here about the leaderboard here on XJS, the agent harness itself may of course also impact the results here. So you can see Kim K3 here is being used with the open code uh agent. Let's actually try setting that up in case you haven't tried it out.

[00:15:33]It's actually very easy. So I'm going to use open code here. And to actually connect it to the model, I'm going to use the forcell AI gateway. It's actually nice cuz I don't have to sign up for some other service. I can quickly test between a few different models. So in case you also want to try something like that. Well, how I got it to work was basically just run their commands here. You can use mpm if you want, but basically you can just copy that here uh in the terminal. I can just install open code. It's already installed for me. And then I can just run open code. Here we go. So now you need to connect a provider. So you can do uh forward/connect and actually do offer their own subscription these days. But uh here there is an option for forell AI gateway. So if you press enter, it's going to ask you for an API key. Here in forcell I can simply go to AI gateway.

[00:16:18]And then here under API keys uh you can set up an API key. You can even specify spend quota. I really like that. Right.

[00:16:24]I don't want to have any surprise uh bills. Even expiry. I like setting it to 7 days, you know, just to make sure it expires quickly. Uh quote refresh. Uh so actually nice uh API key settings. And then you just plop it in here and press enter. And now you can chat, you can pick the model. So then uh you can check the models that are available. Well, you can simply pick Kim K3, right? And that's how you can now use Kimik K3 here for building XJS apps. And then the next thing I would do if you're going to use superbase here to build full stack app is use the superbase agent skill. So I'm going to copy uh this command that they show. I'll link to this in the description. They also have a cloth code plugin. If I just control C out of here for a moment, I can add the skill. In fact, I can just add the all flag here.

[00:17:08]So it will do that for all the detected and compatible AI coding agents. All right. So now you can see it installed two skills. So there are some postgress best practices itself that superbase recommends. Um so we get those but also on the superbase products. So if we check those out we can find that here.

[00:17:23]So now if I open this we can see uh superbase we can see the skill.md file itself which should have those essential or critical information uh bundled in here. So here you can see it specifies never expose the service role uh prefer publishable keys for front- end code in next.js any next public environment variable is sent to the browser. Right?

[00:17:44]So now our AI coding agent should be aware of this without having to make additional uh fetch calls to get information from the docs and the same goes for uh role level security and views right so views bypass RLS by default right important uh security related matter there and I know a lot of you are confused by role level security and you may have already tried sort of offloading that work to the AI coding agent. Make sure you optimize the AI coding agent setup here so that it is as informed as possible. And remember, it can forget to make uh fetch calls to the docs, right? That's why it's bundled in here. So basically, we want to bundle as much critical information inside the actual project itself. So now if I use open code here and I ask Kim K3, can you tell me can you tell me about RLS that is described in this project? And here now uh it's giving me an output here. it actually is able to uh sort of echo the security checklist uh traps never use metadata and RLS policies views bypass RLS by default. So now indeed it is aware of these issues. So yeah, we have a strong model now. So yeah, I think this is a better setup for building full stack apps. So I think we should try to bundle as much critical information as possible inside the project itself because you can see a lot of these models they have a massive improvement here. if the additional information is bundled with the project and then you don't even need to use the frontier models. You should be able to get perfectly fine performance with uh some of the older models here. That's just one thing we can do to improve our setup here for AI. In the next videos, we'll explore some of the other things that we can do. For now, I think this is good to know about. Make sure to check out the superbase agent skills. I'll link to it in the description and actually recommend that you read the post here as well because it shows how these AI coding agents are interacting with developer tools like a database, like the Superbase products here, how they've been able to optimize that. That's it for now. See you in the next videos. But