What we can learn from Cursor's SQLite Rust experiment

[00:00:00]SQLite was rewritten in Rust. And I know we just hit the button rewrite in Rust.

[00:00:04]And you may wonder why is everybody rewriting everything in Rust, but this is not about Rust. This is not even about SQLite. I am aware that there is the Terso database which already is a modernized re-implementation of SQLite and Rust. That is the one you want to use if you want to use a productionready SQLite Rustbased database. Instead, this experiment, mini SQLite, which is linked below, which you can check out, is not about SQLite or Rust. It's instead an experiment by the cursor team, which is all about agent swarms and engineering a system of AI agents and finding out what works and what doesn't work that can build something like SQLite just from its docks because that is what that experiment is about. There is a super detailed very interesting blog post and we'll dive into that. There are a lot of interesting learnings in there about which we got to talk which you also find linked below where they explain how they ran this experiment and the starting point and the idea behind that experiment was to take the SQLite documentation which is in the end 835 pages if you were to put it all into one document which is of course primarily written for humans. I mean, it's not the most approachable documentation I have ever seen, but obviously it's meant for humans because it's way older than all the AI agent stuff. Nonetheless, it also kind of acts as a super detailed spec, a superdetailed specification because it describes in detail how to use SQLite and what the intended behavior or what the intended features are.

[00:01:49]And the cursor team took that documentation and then they took a public test set, the SQL logic test, which is a a bunch of tests that test the behavior of SQLite or that test queries specifically. and they used that to check whether the implementation their agents built based on that documentation actually works with that official or with that huge test suite.

[00:02:19]Now, a couple of important words of caution up front. This test suite here is all about testing queries and query behavior. It does not test all the features, all the capabilities SQLite has. It does not test performance in general. It does not test concurrency.

[00:02:40]There is a lot of stuff in SQLite which this does not test. And this mini SQLite result of that experiment, one of the results they actually rebuilt SQLite multiple times with different agent combinations and we'll dive into that is therefore really just something to explore. It's not production ready. It's not what you want to use. It's just the output of an experiment where the goal was to use the documentation and then rebuild SQLite and make that rebuilt version pass all those tests. And the cursor team used various model combinations here. Why combinations?

[00:03:18]Because as we'll learn, they used an approach where multiple agents worked together, planner, worker, and reviewer agents. and they rebuilt that SQLite database with different combinations here and then also measured for similar quality. So all these combinations achieved the same kind of quality, the same amount of tests passing, but they measured which combinations cost how much. So for example, GPD 5.5 used for everything for planner agents and for worker agents led to an implementation cost of SQLite in Rust based on a documentation of around $10,000. On the other hand, combining opus 4.8 with composer 2.5 and composer 2.5 is that super fast, very cheap, very efficient, but not super intelligent model by cursor. Combining these two led to the same kind of quality, the same amount of passing tests as these other combinations for only a fraction of the cost for only $1,300ish.

[00:04:26]And the idea here was to use Opus 4.8, which is the more capable model of course for planning, for designing the tasks, which are then handed off to those worker agents which are using composer 2.5. And that already is one important takeaway from this article, which is not brand new. Of course, you can do that yourself too. If you're building software, it's a good idea to split your work, depending on the complexity of the work, of course, into different tasks that are executed by different agents using sub agents, if you will, where some agents focus on the planning, on the signing a focused task of the overall task. uh so an individual job you could say and then having dumber agents that implement that task because it turns out that for just spitting out good code you don't necessarily need frontier intelligence if the context is good so if the task is clearly defined if all the useful information is in that task description and then of course other factors can matter too for example it can matter how the surrounding codebase looks like how examples look like you provide to an agent that all influences the output. But worker agents that just write code can be dumber if the task is well specified and the context is good. And that's what most of the experiment was about. how to design a system that can tackle a task of this scale because of course as I mentioned for your projects as well it is worth a consideration to have planner plus worker agents now obviously not for all tasks if you have a quick bug fix if you have a very simple task it's absolutely fine to just tell your agent codeex clot code whatever it is hey I have this problem I want you to do this give it some extra context and let it do its job. It may depending on the coding agent harness spin up sub agents nonetheless. Clot code may do that.

[00:06:32]Other harnesses like pi may not do that if you don't give it the right extensions. But even without sub agents, many tasks can be tackled by just one agent and you'll be fine. But for more complex projects, more complex tasks, having that split can be useful, including reviewer agents. That's something I personally also like to do.

[00:06:54]Again, depending on the complexity of the task, but having that split is something that works really well. And that's of course not groundbreakingly new. What is new is that for something like that SQLite rewrite here, you have multiple parallel processes of planner, worker, and reviewer agents. Multiple workers, but also multiple planners and reviewers. And they clash all the time.

[00:07:20]That is what the cursor team in the end found out here. Now in this blog post which is very very interesting. They mentioned that earlier this year they already ran an experiment with an agent harness where they built a web browser from scratch and now they used that same harness or that same system of agents I should say to do that SQL light rewrite but they also built a new system simply based on learnings they now have as an experiment. And then in this experiment and in this blog post they compare these different approaches and dive into all the challenges they encountered whilst trying to set up and run that system that rebuilds SQLite. And one of the first challenges they faced when working with a system where multiple many hundreds or thousands of workers work simultaneously is that traditional version control Git doesn't do it anymore. as they write.

[00:08:15]Um, in an earlier post about the swarm, we note that the tools like git and cargo rely on course locks for concurrency control, meaning that the same piece of data is being locked so that it can't have multiple writers simultaneously.

[00:08:30]This is fine for one developer, but unworkable for the volume of work produced by hundreds of concurrent agents. The browser swarm from earlier this year peaked at roughly 1,000 commits per hour. So that's the swarm that rebuilt that browser. The new system, which they designed for this experiment, peaks at around 1,000 commits per second. So the old swarm, which they ran earlier this year, had 1,000 commits per hour, which is a bit more than most humans have, of course.

[00:09:02]Obviously, the new system, however, had around 1,000 commits per second, which is mindblowingly much. and clearly clearly not what git was built for.

[00:09:15]Obviously to facilitate this rate of activity, we built a new version control system from scratch. Throughput was not the only reason to own this layer. Every change in the system passes through the version control system. So it's where collisions first become visible and several of the coordination mechanisms in the next section are implemented directly inside of it. And that's really interesting.

[00:09:39]They built a new version control system for the AI agent age because the old one git which we all use of course and there's nothing wrong with it. Just to be very clear we're talking about an experiment here uh at a scale and on a task that many of us will never tackle at least not soon. But still the old system git is not built for having hundreds of agents, hundreds of entities working on the same code simultaneously.

[00:10:08]So they built a new version control system which can handle insanely high concurrency but which can also help with resolving conflicts via agents because obviously it's in the version control system where conflicts become visible.

[00:10:24]if two changes affect the same piece of code in a file. So that's the first important thing here. They built a brand new version control system for this experiment to be able to efficiently run it or perform this experiment. Now naturally as they mentioned here, they encountered many problems at that scale and rate of change at 1,000 commits per second. for example, and this is really interesting all these issues they encountered and how they solved it because it gives us a glimpse at how software engineering may look like in the future at least in in certain scenarios. The split brain design problem, two planners unaware of each other implement the same concept in different ways in different parts of the codebase. So duplication, same concept in different ways in different parts of the codebase. You typically would want to extract that and reuse that logic, right? We fixed this through prompting.

[00:11:25]So no fancy new system built here.

[00:11:27]Instead, through promp prompting planners make the sign decisions, the planner agents themselves rather than delegating them. And we require them to ensure that no two delegated sub trees decide the same question. So it's a setup question here. It's all about ensuring that as you split that system into planners, workers and so on, you ensure that your different planners, because it's not just parallel workers, it's also parallel planners have clearly defined tasks that are highly unlikely to clash and overlap. That of course starts with the design from humans. So how you set up the task, how you prompt, right? We fixed the this through prompting. But that of course then kind of goes down that tree of agents with all those parallel nodes and all these parallel leaves where you want to ensure that as you have agents split up tasks into subtasks, those agents are prompted to cut tasks that have a a low chance of overlapping. So that is ultimately a planning challenge for humans which is all about setting up the system in the right way from the human side. So that's how they solved or how they tackled this problem. Another problem they faced was contention between planners. A harder form of contention is when two planners know about each other and fight through back and forth changes over the same files. The problem is two pictures of reality and merge tooling can't fix a disagreement. Instead, we have agents record decisions in shared design docs.

[00:13:08]Code that depends on a decision carries a compiled check reference back to its dock. When planners unknowingly contradict each other, a reconciler merges the docks and the references propagate the resolution downstream. So in the end related to the previous point when splitting work across planners still of course naturally in software development you can't entirely avoid overlaps or shared domains shared logic shared areas that need to be touched by planners and ultimately workers in a codebase. So that is when planner agents started to fight over an implementation and they fixed that by bringing in a reconciler agent I assume that merges the docs that were crafted by those planners. So that were crafted to then hand out to the workers. They had a reconciler step in between that merges those docs of fighting planner agents so that they would talk a consistent language and agree on an implementation to also help with ensuring that the same thing wouldn't be re-implemented in different ways in different parts of the codebase. So these two work together as I understand it. Now naturally they also encountered merge conflict. So planning and ensuring that there is no overlapping there or as little overlapping as possible and that planners speak the same language is the first important step. But still multiple workers even multiple workers working on one and the same plan are of course very very likely to touch the same files to contradict with each other and to create merge conflicts within the swarm. Agents constantly collide on the same files. In order to resolve a collision, they would have to stop, absorb the other agents context and merge around it. Naturally, if two agents or two humans for that matter work on the same file in order to resolve that conflict, both have to stop. No matter if it's agents or humans or normally they would have to stop so that you find a a decision, an implementation that resolves the conflict. Worker agents, however, are bad at this. and in practice either overwrite the other change or abandon their own. And maybe you noticed this as well. I certainly have. If you work in a codebase together with one or more AI agents and you make a change, okay, I know it's scary, but you can still write code. So let's say you make a change, you change something in code, the agent will always just undo it and overwrite it. It does not respect those changes.

[00:15:50]It has its agenda and if it decided that it must edit that given file, it will do that and it does not care if you made any change to it in the meantime. It's a bit different if you committed that change because these agents are post-trained, fine-tuned to not easily undo your commits and stuff like that.

[00:16:14]But if it's an uncommitted change, the worker just doesn't care. The agent just doesn't care. And that's exactly what they encountered here too. To fix this, we created a system where a neutral third-party agent intervenes on merge conflicts and resolves them on behalf of all parties. Its only goal is to be impartial and efficient similar to the way merge cues work in engineering teams. And I think that is also interesting. It's again a form of reconolation. It's again, as I understand it, about stopping those agents just like in the old world where you had to stop and take a step back and find a resolution for a conflict. But what this clearly shows and that's also not a new learning is that fresh context with the right context given though is super super important. So no matter if you're tackling a large scale task like cursor here which we all don't do or if you're just working on a smaller scale project the huge advantage of having a split across planner and worker and reviewer agents is primarily or very often at least that you work with fresh context windows. This does not mean that it's empty context windows. just means that you have fresh agent sessions populated with just the right context for a given task. For example, a worker is pretty bad at reviewing its own work because it has all that context from implementing that stuff in its context window. So, it's biased if you want to call it like this. That's why a reviewer agent should start in a fresh context window should get the information what the worker worked on, what the plan was, which files were touched, but nothing else so that it can honestly review that work. That's why fresh context windows populated with the right context are so important. And that's exactly the same thing here where they resolved merge conflicts through a new agent with just the right context without having a bias that could then resolve a conflict. And then I assume uh the system was set up such that those worker agents received information that this is the conflict resolution and they must not override it or new worker agents were started.

[00:18:30]That's not entirely clear to me here.

[00:18:33]Another issue they encountered were mega files. Some files are particularly popular places for agents to work. Each agent might add only a small amount of code and no single agent is responsible for keeping the files small. These mega files choke everything. They're expensive to transport, diff, merge, and become the site of constant collisions.

[00:18:55]Again, that is also something which on a far smaller scale you may have encountered as well. I certainly have.

[00:19:02]especially for testing. My experience is that agents love to just add more and more tests in the same file. And it's of course not just testing, but that's one area where I can frequently see it. And especially if you have multiple agents working and each agent has its agenda, they don't care because they're not humans. How could they care about anything? They're just executing tasks, right? They don't care about the size of a file or the general architecture of a system. If you just have a bunch of agents executing their tasks, your codebase will drift into chaos at some point because agents don't care. They care about executing their task. And these mega files of course are one clear indicator of that problem that they become a thing the more agents work uh for a longer period of time in your project. there is no agent there that's responsible for splitting that file or for keeping your codebase well architected. That's just not their agenda. So to fix this, we gave worker agents a way to flag bloated files. Once flagged, we block new commits and then outside agent decomposes the overgrown file into smaller modules. So again, a fresh agent coming in. It's a pattern we see here. For all these problems, it was about identifying a problem and then using fresh agents with the right task given with the right context given to resolve that problem so that then the other agents can continue their work.

[00:20:36]And that's the same here for the megapiles oification. Another problem they faced agents have learned from working in existing code bases with humans in the loop not to touch core code even when it needs to change. So this is not what I meant before when you make a change in a file the agent is working on and it just throws that away.

[00:20:56]It's instead about in general a agent has a clear task based on a plan based on a prompt you gave it and that does of course involve changing certain files.

[00:21:08]Now, one thing we already know or see uh every day when working with agents is that depending on the model, some models are super hesitant to let go of existing code. They rather add 10 fallbacks, 10 if checks, and more and more legacy go code to a codebase instead of deleting it and cleaning it up. you have to explicitly prompt to make sure that a agent really deletes a function or gets rid of some code file. They don't really do that on themselves because of fine-tuning because clearly these model providers don't want to build models which then roam freely and break all kind of production code. But when not working in a brownfield project, when not working in an existing codebase that's maybe running in production, then this tendency to not really touch code and to keep all the code around forever can be super problematic and annoying.

[00:22:05]And it can also lead to other side effects like they encountered here where agents would just not improve code written by other agents but just build on top of it over and over again leading to a bloated codebase ultimately. Of course to fix this we license intentional breakage. An agent that judges a core change worthwhile can make a focused patch outside its scope and leave a comment explaining why it did it. And that's again on a smaller scale what you can do in your projects too.

[00:22:40]What I'm doing, you want to explicitly license and tell your agents, hey, we're building this. We're in early development. This is not live yet. I want breaking changes. So clean the code up aggressively. refactors are welcome.

[00:22:57]Stuff like that. You want to encourage agents and and these AI models and overwrite their fine-tuning instructions, so to say, their built-in knowledge and get rid of that to make sure they can indeed evolve a codebase instead of just adding more and more code to it. So again, something we can see on a smaller scale here of course seen on a large scale. Now for reviewing they used an approach uh called review lenses. So we have the the planner and worker agents but of course that work must be reviewed to then create follow-up work and and restart the loop until a certain bug is fixed until the codebase is in a better shape. In a system that is both longunning and multi- aent errors accumulate and the swarm needs a way to correct itself before small mistakes become foundational. Again makes sense. We've all seen this on a smaller scale, too.

[00:23:51]We experimented with many kinds of review lenses, such as giving a review agent the worker's full transcript or only its output or nothing but the codebase. We also tried reviewers running on different models with different training and a different personality. No single lens catches everything, but decorrelated lenses stack the way self-driving systems reach above human reliability without any single perfect component. The compute spent on review is high return since review is much cheaper than the work it audits. We suspect this stacked review system was a major contributor to the sustained quality of the runs. So key takeaway here, it's impossible to have one or more uh reviewer agents each review the entire codebase. It's it's way too much. Instead they experimented with different approaches like giving it the full transcript or only the output or nothing but the codebase. And what they find out in the end is that what helped them is to have different reviewers with different personas with different um focus lenses where they would focus on different aspects. Give them the code base as I understand and maybe some information about what the cur the worker did. And then it was the combination, the output of multiple reviewers combined essentially that led to an overall review result that then could be picked up by a planner again to turn it into a plan and have workers fix the code. And again on a smaller scale I I think that is something you can or we can apply too. Now obviously um we're again we're not building stuff like that but what works really well in my experience too is to have multiple reviewer agents with different tasks where one of them may focus on hey is this ideatic rust another one may focus on performance and security issues if fable 5 lets you another reviewer may focus on naming patterns if that's something you you want to focus on and so on So you have different lenses and then you give those reviewers just the right context. Again, something like, hey, we had workers work on that feature. Maybe give them the plan of the of the worker and then maybe some information about the the rough steps the worker did, but nothing more. And then you have all this output from the different reviewers and you can combine that again maybe with another reviewer.

[00:26:23]What I also like is have a reviewer review the review results.

[00:26:28]uh which simply has the idea that depending on the model again they tend to like to find stuff no matter which code you hand them. It could be a oneliner I sometimes feel like and they would find five issues in it. So have a reviewer categorize those review findings and drop the ones that aren't really issues can work really well in my experience. And it's this again as they write combination of reviewers, this stack of reviewers that can help produce good results which then can be picked up and implemented again. Now again always depends on the scale of your task of the software you're building. Obviously for for many many pieces of software this is all way too complex but it's a nice glimpse into how software engineering could look like how such systems could look like in the future which I personally find very very interesting.

[00:27:27]Now uh one last thing they also did is they let agents shape the environment here. The idea was that they let agents write a field guide in the end a document or a collection of documents uh where they did not give the agents any instructions other than that this field guide should act as context for the overall task where agents could therefore build memory shared memory you could say um which is all about learnings or key issues that maybe were identified and and things like that. um so that they had this extra memory system if you want to call it like this in place for the agents to keep notes and document decisions and overall I find just like the bun rewrite in rust I find this experiment very very interesting it can also be scary I totally get this and I think we should not infer that this is how all software should be built from now on I mean for one this is not even a production ready piece of software and getting it production ready would certainly certainly take a considerable amount of time. This is not to be underestimated.

[00:28:36]It's not like you can build something like this in in in a couple of hours and then making it production ready is only a few hours more. The first 80% can be way faster to to achieve than the last 20%. We all know that. So that's one important takeaway. It's also important, of course, to realize that this specific task of rewriting SQLite, yeah, it may just have received those docs and then this test suite here was used. But obviously, for one, this is an incredibly detailed spec here, something you don't have for new projects. If you're building a new piece of software, you don't have a specification as detailed as the documentation for a software that that's over 20 years old.

[00:29:24]And of course, even if only the documentation was given to those agents, the SQLite source code and also maybe other source code like other uh reimplementations like in Rust here by Terso may and and is very likely to be part of the training data of most or all of these models that were used. So it's not like it was brand new for these models. It's not the same as a brand new piece of software being built where iteration is also an important part of building it. You will have a very hard time and I would even say it's impossible to build a new piece of software no matter what it is from scratch without it changing all the time because you can't write a perfect spec up front and then be done with it. You always discover new stuff or things you want to change whilst you're building something no matter on which scale. And therefore, of course, this is not representative of how software will or should be built in general. It is a very interesting experiment though. It is a very interesting experiment and it has key learnings that matter for all of us. key learnings that of course aren't brand new like splitting your work, having fresh context windows with just the right context. Interesting insights like that maybe new version control systems will emerge and will be required in the future. And of course that multi- aent orchestrations become a thing. Now that all of course also proves that humans designing these systems, these agent systems and humans of course for for new software especially also making decisions on how that software should be architected.

[00:31:23]writing spec files even if they're not as detailed as this and coming up with software architectures and then building agent systems that can implement them and then reviewing that which is important.

[00:31:34]This is where we're all heading and which even though it's all changing and definitely not how we built software six years ago is something that gets me excited. I I think it's it's really interesting that we're moving into that systemic thinking area both for building the agentic systems as well as designing the software architecture itself and then we have both work together. I find experiments like this very very interesting. The learnings here are very very interesting and some of these learnings on a smaller more simplified scale may matter to everyday software and development projects as well. But as always, let me know what your thoughts are and what you think about experiments like