Stop Telling AI It’s a Senior Engineer

[00:00:00]One of the techniques that I've heard many people doing when they're prompting their LLMs or claude code is to add a prefix such as you are a senior engineer to do a little bit of roleplaying so that the LM can actually think a little bit better and be smarter. But I've never actually seen tests to verify if that actually improves the output of the code or the designs or if that's just a bunch of extra noise that confuses the LLM. So that's exactly what I decided to do. I created an experiment to test five different features that I'm going to add into an existing codebase. Five of them do not have the prefix you are a senior engineer, but the other five do have that prefix. And then after building out all these separately in different work trees without having any skill set up, I have a Fable 5 agent going through and reviewing all the code and rating it from 1 to 50 to determine which one is actually the most senior engineer type of code. So by the end of this video, we will learn three new things. The first thing we will learn is adding this prefix. Is it actually useful or is it a waste of your time? The second thing we'll learn is when adding it, what in the code actually changes or improves versus when not adding it. And the third thing you will learn are the skills that I use that actually force these agents to think harder, audit better, and create actual senior level code. So, one of the key things of this experiment is I wanted to run these on real code bases. I didn't want to just spin off a little side project or have it build something from scratch. I wanted to see how well it can add in features to an existing codebase because honestly that is where you're going to be using these agents the most. Most of the things you see on Twitter and YouTube with people building something from scratch. You don't do that in the real life workforce. Okay. And the experiment is like this. We're going to ask Fable 5 to think of five new features to add to the codebase. I am going to keep the features somewhat small but not too small that it's kind of a pointless test. So basically I have five different prompts to add in new features to a codebase and then I prompted it to create 15 isolated work trees. Now the reason I have 15 is because there's also a third test I'm doing which I'll share at the end using my own skills. But I want to kind of focus right now on the baseline in the senior engineer prefix and how it kind of works. Okay. So for every single thing that cla code worked on I told it to create a separate isolated work tree. Spin up a claude code instance. Do not load any skills.

[00:02:18]There's actually a flag you can provide to claude code to not use any skills.

[00:02:22]It's called disable slash commands. I told it to use opus 4.8 with effort extra high. Obviously, if I use fable, it would have burned through my entire 20x max plan. So, I need to actually get this experiment done because this is a lot of things running at once. So, again, I used headless cloud code instances in parallel. I didn't want it to use any of my built-in memory systems or anything else. Just use cloud code headless. Run the prompt and see how it does. Now each instance that's running does self-verify a little bit. So it does run some test suites. It runs the llinters. It does run a production build to make sure that the code that it added in isn't completely broken, but I didn't have it run full into tests or anything like that. Right? So just a an initial run through to make sure that the code seems like it's pretty good. And then I told it to commit and open some pull requests. But the important part is the blind judging by a stronger model. So the implementation and the planning for each feature was done with Opus 4.8. But at the end, I told Fable 5 to go through every single pull request and grade it based on various technical analysis of coding. Is it decoupled? Is it secure?

[00:03:27]Does it seem like it works? And then I had it generate an entire report for me that we're going to show in this video just in a second to see which one performed the best. Now, some things I do want to point out is that they all started with the same base commit. It's good to have a starting point so that the test in the experiment is repeatable. We made sure that we used Opus 4.8 eight with effort extra high for everything. The same prompt for every single request just with that little prefix in front. Isolated work trees like I mentioned, headless, one shot, no skills, one judge, one rubric.

[00:04:00]So, I tried to keep this as fair as possible, but you guys can leave in a comment below if there's a better way I should have ran this experiment. Let's also look at how I actually prompted Claude Code to run the testing scenario.

[00:04:11]I didn't have to code anything. I let Claude Code figure this all out. The first thing I did was I asked it to look over my codebase and give me five smaller size features that are at least worth running this experiment over and it gave me the five that it thought were good. So pre-run estimate costs which is basically how much will stuff estimate the cost before I run it in this application I showed. This one honestly is a very loweffort easy. I think any model could probably do this perfectly fine. a notification system so that when runs finish I can get a not notification and maybe even a bell icon I can click to get a dropown of all those finished things run tags and labels. So this one's a little bit larger. This is a feature that adds tags to all my runs so that later on I can go and filter and search for them if I need to. And then finally the last feature is a one-click run again feature. So after you run a job, let's say you want to kick it off again, click a button. Does that properly hook into all the existing systems and code modules to kick off that run for a final time? Make sure you do these in separate work trees. Do not ask me any questions while you do this.

[00:05:14]I've noticed that sometimes when you run these experiments, the model will try to get user feedback and then it kind of breaks the experiment because obviously I'm not sitting here giving different answers to the different models because that would definitely throw off the results. Do not use any skills. And then at the very end, I told it to basically review the changes using Fable 5. Okay, so that's the setup. That's how I kind of got the whole test scenario set up.

[00:05:38]You can use Cloud Code to do most of anything that you can imagine. So your lack of creativity is honestly going to be the gatekeeper for you to become very excellent at these tools. If you can imagine it, you should probably just prompt it in the cloud code because it could probably figure it out for you and build out a testing scenario suite better. Now, the reason I picked Fable 5 for the whole orchestration is because it's much better at doing these type of things and keep track of all the different experiments. It's making sure that it's running them properly and consistently. And then I kicked off the 4.8 workers just to try to save on a little bit of cost. Okay. So, what is the actual outcome of this experiment?

[00:06:15]Is it worth adding that prefix think like a senior engineer or you are a senior engineer? We're going to find out right now. So, let's go over to the results. All right. So here are the output results. So basically we have the baseline which is going to be in the blue. This is the the run or the test suite that did not have the prefix you are a senior engineer. And then we have the green which is adding that prefix in. Now after the five runs fable 5 judge these and you can see here the green is actually a little bit lower on most of these runs. The the green is the one with having the prefix and the run tags feature did very poorly. I think it actually had two bugs or two crashes that happened. The perr run cost also performed the very worst at the very end. And kind of the blue ones, the baseline actually was more higher at the top. If you were to draw an imaginary line halfway through, it looks like there's more green at the bottom than there are blue, which is pretty interesting cuz I thought actually using this prefix could potentially make the scoring be a little bit higher. Maybe it could trigger the LM to think a little bit harder. But I guess its idea of a senior engineer probably adds a bunch of noise. So let's go down and read through why some of the baseline wins. So the oneclick rerun both endpoints are near identical and correct. But it says the test decided baseline proved the two properties that define the feature. The profile drift test shows that the frozen plan wins over the mutated runner profile and a concurrently capped test showed admission gates reapply. But of course there's other features where it marked that the senior uh prefix actually won. So it could actually be the case that in some scenarios some features do perform better adding in the prefix. But of course we're dealing with non-deterministic LLMs. So overall is it safe to say that this is a clear do not add the prefix? Probably not. As you can see here the run notifications got three points higher than over here the run notifications. So, at the end of the day, is this statistically significant to say that, hey, adding this prefix is actually useful or not? I would probably say no. I would need to add a bunch more features, a bunch more experiments, try doing some with bug fixes, and also try just doing reruns, right? These are non-deterministic machines, so rerunning them a couple times could give us a better average. And maybe all this is actually kind of pointless. Which leads me to how do I actually make these LLMs perform better? Because if adding a prefix to the run doesn't really do much, what is the approach? And honestly, the approach is to spend more tokens and to have these LLMs re audit the code that it writes. So, this is where I'm going to show you the skills that I typically use in my day-to-day coding job at work and also on my side projects. I don't want to have to come back and fix the bugs of the LLMs. I want to be able to prompt it once and come back when it's done and have certainty that it actually works. So, I do have my own set of skills. If you go to agentsystem.dev/skills, these are free, open- source to use. But what these skills do is based on what the request is inside your codebase, like if you're adding a feature or removing a feature, it's going to basically conditionally run a different set of checks and audits. Especially if you were to change, for example, backend code, you might have more security audits that kick in automatically. If you're changing front-end code, you're going to have accessibility audits that kick in. You're going to have pi audits kick in to make sure that the front end is actually accessible. But from running this experiment with the skills versus the baseline, you can tell clearly that using these skills, it comes out on top every single time. This is one in every single of those five features. And they score significantly higher in many of the different categories from the baseline and the ones with the senior engineer skills. Okay, so scrolling down, you can see the full ranking here.

[00:09:51]Every single time I use the ship skills, the Fable 5 judge ranked it as the best versus the baseline and the senior.

[00:09:58]Obviously, they aren't as performing as well. Now granted, I think the reason why these are performing better is because I'm using more tokens. I'm telling these LLMs to spend more time, think through everything, re-review everything multiple times until the output is actually senior level, right?

[00:10:15]Until it's actually really good code, it's clean. It's decoupled. It doesn't have security issues. And you can see that through some of these features like the percost run, it was actually cheaper than the the senior prefix for some reason. I don't really know why. I would think that this one would have been much more expensive. But like I mentioned, we're non-deterministic machines. You never know what run is going to do. Um how the caching kind of adds up for the search and filter. Also, it was cheaper as well than the baseline. Notifications cost almost twice as much. And I would understand why notifications is a larger feature. It has more complexity, which means that more of my audits would probably have kicked in to make sure that notifications aren't accidentally going to the wrong people, etc. The run tags by far is the biggest feature that we asked to add in which caused almost a three times increase in how much this thing used in terms of tokens. Right? So overall using the ship skills do increase the overall expense by 81%.

[00:11:10]Which to many people watching this might be like well that's just a giant waste of tokens. But in my opinion, the better intelligence and quality you can get from these coding agents, it's worth the extra spend because for every bug that you have in your codebase, you end up spending more time going back and manually verifying it, going back and reprompting the agent to fix the stuff that it messed up on. And then you just keep on doing that loop over and over again. And those costs add up as well.

[00:11:35]We just don't really track them as well, right? We typically will run something, we'll go back and look at the usage, but do you ever go back and actually track after fixing it and reprompting it and making sure it's good, how much money is being spent overall versus just a one prompt, get it completely done. Okay.

[00:11:51]Now, with all that being said, I do recommend to take this with a grain of salt. You may get better results using your own set of skills. You may get better results just by adding a couple of extra keywords. And that is the thing that we need to start doing in this industry. We need to run more eval. need to really understand how to make these models output the best highquality code as possible so that we're not just blindly installing skills without any validation or eval to say that they actually do what they're supposed to do.

[00:12:17]All right, that's about it. If you guys want to go check out my Agentic Jump Start course, I have a course with over 13 hours of content teaching everything I've learned along the way. I am going to go through and revamp this course. I made this course at the end of last year and since then I mean like tons of new models have come out, new features have been released in the clawed code. So I want to do a revamp on this but if you do purchase now I will be increasing the price when I get this revamp completely done. So it is at a lower price right now. Other than that, if you guys enjoyed, leave a comment.