Evaluating AI Agents | How Numbers Drive Real Fixes

[00:00:01][music] If you've been following the series, we built a general coding agent from scratch. The model, the harness, the UI, every primitive by hand. Last episode, we let a model or a loop improve it.

[00:00:17]This episode, I did something different with it. I gave it one real job. So, I play in an amateur cricket league called NA. and the league's stats live in a database. So I stood up cricket ID the same model and harness underneath plus a cricket commentator persona in the spirit of hashuk plus a set of readonly query tools over the league data. Ask it who scored the most runs in the league and it names the league stock scorer straight out of the database. You ask it for the individual highest score and it says 124 with full commentary box confidence but the real answer is 152.

[00:00:59]Both of those answers came out of the same agent and the second one is the reason why this episode exists. I think this narrow shape is where agents are going. Enterprises and builders like me are going to spin up small agents that know one domain, one database, one workflow. And a lot of us will use a general purpose agent to build that narrow one which is exactly what happened in this case. The from scratch agent from our series is what cricket is made of. So what this agent really is let me show you the build because it's smaller than you would expect. Gemma the agent from the series ships as separate packages. The model layer behind a provider seam and the harness the drive loop registry tracing policy. In this case, Cric it forks none of it. It's a thin layer of configuration that is handed to the same agent class. A system prompt, a set of tools, a readonly policy. And the whole assembly is just about 10 lines. And this basically here shows those almost 10 lines. And you can see the agent come into existence in stages. Stage one, just the system prompt, the hers persona or the bear model. In stage two, we register about 15 deterministic tools and a guarded fallback. We ask this question and now the transcript shows a real tool call event, a top run scorers. And the answer comes back from the database. The same top run scorer in the same 275 runs from the opening slide that we saw. That is the entire agent, a system prompt tools and a policy writing the entire harness that we uh built up until now. For the rest of this video, this is the thing under test this very specific agent. So if last episode was about improving uh the hardness uh and hence the agent, this one is about measuring one and I built an evaluation harness around cric it ran it and I'm going to walk through what it measures what it found and what I'm changing next based on actual numbers. And if you are building your own agent or shipping an AI feature inside a product, my hope is that you can lift this whole approach for your own eval. The vocabulary, the metrics and the way the numbers turn into next steps. Now, a general coding agent is hard to grade. Write me a script has a thousand acceptable outputs and deciding whether one of them is good is a judgment call all the way down, right?

[00:03:41]Um, a domain agent is different. who has the most runs has one correct answer and it's sitting in the database. I can query myself. So the correct behavior is definable which means it's measurable.

[00:03:54]Correct answers are only one slice of good. An agent can also be right but slow, right but expensive, right for the wrong reason or wrong in a way that's actually fine. So before any numbers I had to decide what good means for this specific agent and it came out to six different things but before those let's focus on the vocabulary because eval world has one and it's simpler than what sounds like. So the mental model is five words and I want to define them in everyday language because everything else in the video hangs off of them. So I built a data set. A data set is a collection of items.

[00:04:42]Each item is a question I would genuinely ask cricket. Who scored the most runs? What's the highest individual score? How many centuries have been scored? For each item, I decide which metrics I want to measure. Did the answer match the one in database? Did it call the right tool? How long did it take? How much did it cost?

[00:05:06]And every time I run the evaluation, each of these metric produces a score.

[00:05:13]And tracking those scores over time is how we'll tell whether a change that we make to an agent actually improves it or just feels like it does. Okay, so my baseline data set is 25 items and I run each one three times on the same local GEMA 4 26 billion A4B model that is powering this same agent and the repeat matters because the same question may not come back the same every time and a single right answer can be a fluke. So if cricket nails a question on the first try, the second and third are what tell me whether that was skill or that was luck. The repeats are where the confidence in the measurement comes from. Be specific. Here is one item from the data set. All five lines of it. The ID which is H05 in this case. The question what's the highest individual score in a single innings? Its category in this case it's happy. the kind of answer I expect a number in this case a little function that computes the correct answer from the database itself the eval world calls that the gold answer and because it's computed I never hand typed an expected value anywhere and the tool I expected to reach for next to it everything the hardness recorded when the item ran the answer it gave on all three tries 124 um zero tool calls the answer contract intact and 2,162 tokens per attempt one item and you can already read four different kinds of verdict off of it. So with that machine in place, the question is what to point it at and I landed on six. Did the answer match the database or the answer in database? Across the baseline, cricket lands at 0.789, which means roughly 79% of answers were correct.

[00:07:22]That's the headline number, but there's a lot more to it than it shows. So trajectory is the second. Which path did it take? When cricket gets a stats question, it's supposed to answer by calling one or more of its query tools.

[00:07:37]And I checked whether it called the right ones. When it reached for a specific tool, it picked up the right one 93 of the time. So a correct tool rate of 0.933. A right answer that came from a guess instead of a query is a failure waiting for a new question. And the trajectory data is what explains the confident 124 from the opening slide. On that question, Crricite made zero tool calls on every attempt. The number don't come out of the database at all because nothing was queried. The model just guessed. Outcome tells me the answer was wrong. The trajectory tells me why and why is the part I can act on. Cost is the third one. Every answer averages about 5,6 262 tokens and about 6.4 seconds. For a hobby league bot, that's fine. For anything at volume, it might be something that you might want to act on. Fourth is reliability.

[00:08:41]Ask the same question three times. Does it come back right three times? That's what we are measuring there. Safety is fifth. The data set includes prompt injections, destructive requests, attempt to fish out personal data and out ofd domain questions and cricket declined every one of them. So safety score one and a part of that is architecture rather than the model itself. The tools are read only and everything runs against a frozen redacted local snapshot rather than the live database. So there is no code path that can modify anything whatever the model decides. And finally it's the experience. Does it stay in character?

[00:09:24]Is it clear? Does it admit when it can't find anything? Does it ask a clarifying question when my questions are ambiguous? That last one produced the worst score in the entire evalu. And that is 3%. And so here's what the whole baseline looks like on one screen. The split the headline by category and the picture changes. Happy path questions 92%age. Adversarial and edge cases clean 1.0. Questions I already knew it struggled with 67%age.

[00:09:59]Ambiguous questions 17%age. And the questions with no matching tool 33. Same agent wildly different behavior depending on what you ask. The ties across the top are the runs whit the answer contract safety cost. Now if you notice the categorization is important here because the correctness or the behavior of the agent depends significantly on what the category was.

[00:10:28]If all you checked for was one happy path or 10 happy path, you are going to get 92% of them correct. The diversity or the categories of these questions are important because that might be hiding some of the information that you might want to know about your agent. Now, if I ask the same question three times, how many times does it come back right?

[00:10:52]That's reliability and it's split into two different questions. The first is for how many questions did cricket get the right answer at least once out of the three tries for this baseline that was 84%age and the event literature writes that as pass at K which means pass in our case at three how many questions of all my data set did cricket get the right answer at least once in the k tries or three tries the second one is stricter For how many questions did it get the right answer every single time? In this case, that's 76 written as pass to the K. The gap between 84 and 76 is flakiness. Those are questions KIT can answer and sometimes won't. If I ran each item once, I'd never see the gap.

[00:11:50]And that gap is the difference between an agent that knows something and an agent you can rely on to say it.

[00:11:57]Flakiness also isn't spread evenly. The questions served by a deterministic tool mostly pass every time. The gap concentrates in ones where the model has to answer by creating its own query. Now remember there are deterministic tools in my agent and there is a fallback where if it's not one of those tools it tries to create the query by itself and that's where the gap was concentrated on. We broke every answer down by type of the graded answers 59 were correct 11 confidently wrong and five where it declined telling me that it couldn't find the answer. The confidently wrong fraction works out to a hallucination rate of about 15%.

[00:12:49]Or.1 the 11 are the ones that worry me.

[00:12:53]The higher score question is one of them 124 delivered like a fact when the real answer is 152. Another one was how many centuries have been scored in the league? Kati says zero and there are plenty. So a wrong answer delivered confidently is dangerous precisely because the user believes it. Nobody double checks an answer that sounds sure of itself. Now those five declines are wrong in the safe way. I asked how many 50s had been scored. Its query came back with nothing and it said so instead of manufacturing a number. The eval literature has a name for that move and abstension.

[00:13:37]The model choosing to say it doesn't have the answer rather than guessing one. So on a scoreboard a decline and a confident wrong answer look the same.

[00:13:48]For a user they are completely different and once you measure them separately you can start engineering towards the safe one. So then let's talk about grading them. on grading. Wherever plain code can decide, I let plain code make the call. Did the answer contain 275? Did the agent call top run scorer function?

[00:14:11]How many tokens did the run burn? Code checks are cheap, deterministic, and they never change their mind between runs. Two design decisions make the code checks possible, and both live in the agent. Every criciti reply ends with a small machine readable JSON block carrying the answers value and its kind with hersa commentary above it. The pros is the persona.

[00:14:38]The JSON is for grading. Whether the block shows up correctly is itself a metric and it held at about 97.3%age.

[00:14:49]The answer key is computed rather than typed. The eval calls the same deterministic query functions the tool use straight against the snapshot and then gets the gold answer. That's how the gold answer is generated. I never hand typed an expected value so the key can't drift out of sync with the data at least in this case. Some things code can't reach. Did cricket stay in its commentator persona? Was the answer clear when my question was ambiguous?

[00:15:23]Did it ask season I meant or did it pick randomly? For those I use model as a judge. I handed the question the answer and a rubric and it grades pass or fail.

[00:15:36]I kept its remmit deliberately small four yes or no criteria with an allowed unknown so it never has to force a verdict. Now two caveats on the judge.

[00:15:48]The first my judge is the same local model that powers the agent. It's grading its own homework and a model will happily flatter itself. So before trusting any judge number, I calibrated it. I handlabeled 24 of the same answers myself and checked whether the judge agent agrees with my labels. If it agrees, the score would be higher. and it agreed on almost all of them across every criteria. The stats shorthand for that agreement is kappa and mine came out at perfect 1.0 which is the only reason the judge scores appear in this video at all. The second caveat surprised me the most. Persona scored 97 percentage, clarity 97 percentage, admitting when the data doesn't have answer 84 percentage, and asking a clarifying question when the question itself was ambiguous, 3 percentage. Ask cricket something vague like who's the best player and instead of asking what best means, it commits to an interpretation and answers with full confidence. Roughly 97%age of the time it's the same disease as the confident wrong answer showing up in the experience layer. This agent would rather guess than ask. Now as I mentioned a design choice from when I built crick it would matter here. The tools are deterministic on purpose. 15 of them. Each one owns exactly one correct query. top run scorers, player batting summary, team standings and so on. The model never writes MongoDB queries for these. It picks the tool, fills in the parameters and the query itself is right every time by construction because it's code. I can't pre-build a tool for every question though. So there's a generic fallback aggregate capability where the model does author its own query allowed only when none of those specific tools fit and the eval harness knows every single time the fallback fires. How does it know? Well, two design decisions at two different layers and together they are the template that I would recommend and suggest if you're instrumenting your own product for EVAs. The first one came free because of Gemma. When we built Gemma, observability went in as its own primitive. So every model call and every tool call already gets recorded in a trace. Crric inherited that machinery unchanged. The second one was on me as the agent's author because a raw trace only tells you a tool ran and that isn't enough to score anything. So every cricket IT tool carries labels the evals will need. When I register a tool I declare which tier it belongs to domain or fallback and the harness carries that label into every recorded call at runtime. Cricket adds that one thing only runtime can know whether the result came back okay. Did the result come back as an error or did it come back as empty. The eval harness lives entirely outside the agent and never has to know cricket's internals. It just reads those attributes. It is agent's responsibility to provide those attributes. Fallback rate is counting calls labeled as fallback. what it is measuring or recording is the question was not answered by any of the existing deterministic tools. So a fall back was used. Recovery is scorable because empty is temp right there in the trace. So if a tool call or a query returns empty that's recorded. The harness gives you the recording for free. But deciding which labels the evaluation will need is the work you would do while building the agent and it's much cheaper there than as a retrofit. In the baseline, the fallback fired on about 15% of the answers. A fallback rate of 0.147.

[00:20:21]Two numbers next to it tell me the routing works. Every time I expected the fallback, Cricit used it and it never fell back when a specific tool existed, which in itself is great. The problem is what happens after it fires. Counting centuries and counting 50s have no dedicated tool. Rakiti drops to writing its own aggregation and it gets them wrong. Which means the eval is doing more than grading the agent. That fall back log is a shopping list. Every question that keeps hitting the fall back and failing is a candidate for the next deterministic tool. Count centuries, count 50s. The eval is telling me where the tool set has holes ranked by how often real questions fall into that. So the eval also showed me the fallback's price and it's steeper than a wrong answer. The centuries and 50s attempt each bonded around 6,500 tokens on their constructed queries and still came back with the wrong answer.

[00:21:32]The worst case was a question about man of the match. On two attempts out of three, the fallback turned what should have been a clean refusal into a thrash.

[00:21:44]Five constructed queries, six tool calls, nearly 29,000 tokens, more than five times the average answer. all to fail. Anyway, the third attempt declined cleanly in a single call. An escape hatch invites over trying and that same instability is the reliability gap from the earlier pass at K above pass to the K showing up in a fallback run. So I still want the fall back since it's the only reason some questions are answerable at all, but it costs determinism. It costs more tokens and sometimes clean refusals and I only know the size of that bill because the eval itemizes. Everything so far is one question, one answer. The harder case is a question that needs several steps in a row. Who holds the highest individual score record? And across that player's career, how many runs and how many wickets? To answer them, the cricket IT agent has to find the record holder first, then look up the same player's batting, then that player's bowling.

[00:23:00]Three dependent steps where step two is impossible until step one comes back.

[00:23:05]And that is a very simple analogy of long horizon task where you have to do tasks that have dependencies and a sequence to it. A note on what's underneath because I went and checked the code before claiming any of these.

[00:23:22]There is no planner module used in this path. Gemma does have an orchestration primitive but cric ID doesn't use it. It runs on the plain drive loop where the model sees each tool result and decides the next call itself. The chaining here is the model's own behavior. And that's exactly why it needs measuring. Nothing structural is guaranteeing that sequence.

[00:23:48]So grading a chain pass or fail throws away the information I actually want which is where it breaks. So I grade with weighted checkpoints. Some credit for identifying the record holder more for the career runs. Full marks for the wickets. Partial credit then localizes the failure. So the result here surprised me in a good direction.

[00:24:15]Cricket changed the highest individual score into player batting summary into player bowling summary and nailed the whole thing on all three runs. The record holder won 164 career runs, six wickets mean the checkpoint reward of 1.0. Put that next to the centuries failure and you get a real capability profile. This agent is strong at sequencing tools and weak at authoring queries.

[00:24:45]That is a far more useful sentence than 75% accurate. One more thing I tested on this chain is recovery. I injected a fault forcing its first tool call to return an empty result and watched what it did. It noticed the empty result. The trace literally marks that call empty.

[00:25:08]retry because I can see in the traces that another attempt was made and still landed the correct answer. The run log records it plainly. Faulted true retr.

[00:25:22]Now the run overtime story stopped being theoretical.

[00:25:27]While I was producing this episode, Gemma was updated with an interface release with cricket's domain code untouched. I reran the same 25 items three times each on the new version. The headline went up from 787 to 813. The mix underneath is messier. The abstensions disappear 5 to zero. The 50's question, my example of the safe kind of wrong, stopped declining and started answering 382 correctly on two of the three attempts. The man of the match question went the other way. All three attempts now thrash through constructed queries nearly 29,000 tokens each and never even emit the answer block. The average answer got about 14 more expensive and the cause wasn't a smarter model. Cricket used to truncate every tool result itself at 6,000 characters before the harness of course clamped it at the door. The new release moves the results budget into the harness. So cricket deleted its own truncation. Same model, same tools, same questions. What changed is the shape of how the model sees when a query results come back. The extension behavior I showed you earlier was never a personality trait. It was the plumbing.

[00:26:56]And that is a really interesting realization. The the agent coming back politely saying that I don't have information and hence I can't answer was never driven by the personality prompt.

[00:27:09]It was actually the hardness around it, the plumbing that made it act. So, so the same 25 question ran on both versions and every score is comparable.

[00:27:20]Tracking scores over time is how you grade your own changes and how you catch behavior drift you didn't ask for from layers you don't control. In this case, I happen to be the harness or the Gemma author as well as the cricit author. But most of the time, you might be using a different SDK. You might be using clawed agent SDK or OpenAI's SDK or Codex SDK.

[00:27:49]And so if those change, your agent can have significant impact as well. That is the point I'm trying to make here. And that means that we need to know what all components can end up impacting our scopes. Which is why in our case it's a configuration version, the model itself, the data set, the the harness baseline.

[00:28:11]So all of those are the ones that we should be tracking and any change in any of those should trigger a new run for our evap. The fix I' had been planning uh handing the agent the database schema now gets measured against this baseline and that is exactly what I did. So let's talk about that. Okay. So here is that experiment that I ran. The failing question failing fallback questions had one thing in common. The model was authoring the queries against a schema it had never seen. It guessed the collection names. It filtered scorecards for per batter stats that actually live in a collection called batting innings and it burned tokens discovering all of that in the hardware on every single attack. The fix used a primitive that has been sitting in the harness since the original series. As we know, Gemma autoloads an agent.mmd file from the agents root folder into the system prompt. crick ID shipped with that loader deliberately aimed at a directory with no agents MD in it to keep the repository instructions out of the persona. So this is the time I wrote one. It's a data map per inning batting lives in this collection. References are ids rather than names and many such things that are added to agents. I also had the exact field names for every collection. Then I pointed the loader at it. That is the entire change. No new tools, no model changed, only one existing primitive switched on with the right content in it. Then I reran the uncovered question against the new baseline. Centuries went from 0 out of three to three out of three and it answers 15. Nah. 50s went from a flaky 2 out of three at 16 to 19,000 tokens to a stable 3 out of three. Both got cheaper, too. One constructed query and about 5,900 tokens per attempt instead of the multi-query hunt that used to fail. Then the full suite, all 25 questions, all questions run three times. Outcome went from 813 to 960.

[00:30:36]And three more questions came along for the right. All downstream of the same blindness. The highest score question, the 124 from the opening slide. Answers 152 on every attempt. Now the man of the match trash collapsed onto a clean one call decline. Best player guess started declining properly instead of picking a name. The evil also flagged the bill.

[00:31:03]One question regressed. How did we do this year? Ambiguous on purpose because nobody said which team we is before the schema map. It held back on two of three attempts. After it, all three attempts confidently dumped the standings table. The richer prompt plus the instruction to prefer a specific tool over every question made the agent more answer eager. It sees a clean path to team standings and takes it instead of asking who V is. A fix shifts the behavior around it as well. And the eval is the only thing watching this entire board. So the second loop ran and I added one paragraph to that same agents.mmd.

[00:31:50]If the question subject is unspecified, don't assume it and don't answer with all of them. Ask confident wrong answer guess is worse than a clarifying question. That's basically the context I gave. Then the full suite ran again and it landed at 973. Every one of those 25 questions right at least once. The ambiguous questions now ask. And the count I care about most confidently wrong answers went from 11 at the baseline to zero. What's still imperfect is one flaky question. man of the match over tried on two of three attempts this run and that single item is the whole remaining gap between got it at least once at 1.0 and got it every time at 96.

[00:32:41]Even that failure is the safe kind and on the attempts where it fails it burns tokens and then produces nothing rather than something false. The judged pass to rescore the experience criteria comes next and the shopping list stands since a dedicated count centuries tool would still beat a well-informed fall back on determinism. But the loop this episode is built around ran twice end to end.

[00:33:09]Watch it fail. Read the trace change one primitive of the harness. Rerun the same questions and watch every number move including the ones you didn't mean to move. Okay, one thing I didn't appreciate going in that evaluation has a bill. My baseline is small 25 items times 3 which is 75 answers at roughly 5,262 tokens each. So call it about 400,000 tokens for the entire evaluation pass and that's a toy data set on a local model. The Eval Eval Coalition, writing on the HuggingFace blog, argues that eval are becoming the new compute bottleneck. The numbers they site around $2,800 for a single run of one agent benchmark.

[00:33:56]So, you have to be selective and the levels are the same vocabulary from earlier. Which agents and which skills deserve a real eval? How many items should be in the data set? How many repeats per item should you have? Since reliability numbers need repeats, and repeats multiply cost. How often should you rerun? I'd rather have 25 sharp items I run on every change than 500 items I can only afford to run quarterly. Where this leaves me, I spent a series building a general agent, and the most useful thing it has done so far is become a narrow one. I think that pattern generalizes domain agents built by general agents one per job each with an eval hardness so the person who built it can trust it and keep improving it without guessing the habit I'm keeping from this whole exercise before I trust an agent I ask what kind of wrong uh it is when it's wrong an agent that says it couldn't find the answer is a tool I can work with an agent that says 124 with a straight face when the answer is 152 needs a harness watching it. The schema map and the ambiguity guard where the first two changes and both are measured.

[00:35:20]Still open are that man of the match overtime a judged pass to rescore the experience criteria on the new baseline and the dedicated tools on the shopping list. Same 25 questions after every change. The before and after in this video are both fingerprinted results and the next ones will be two. This is a good close to the entire series that we have been doing. We started with the hardness engineering primitives. We went deep dive into a few of those primitives. Then we learned how we can create that in practice applying those uh primitives and uh theoretical understanding into an actual agent Gemma. Then we figured out a way to have a loop that does the self-improvement and how that required us understanding of the editable surface. And now once we start using that framework and that agent and that harness, we can create domain specific agents out of it without ever changing the underlying stuff. Idea again is to show that today I use Gemma, tomorrow I might be using PI SDK or agents SDK or OpenAI codeex SDK but when I build this domain specific agent how and what I should be monitoring and observing is what the whole exercise is about. So hopefully you can see this entire sequence and understand the landscape a little bit better when you are building your own agents, your own harness, your own domain specific agents, your own loop. And as always, we'll continue to have material with this. We now have a website which is the carbonlayer.com.

[00:37:03]And each of these deep dive videos is an article on that one with links to this videos as well. Same would be the case for this particular video and series as well. Thank you.

[00:37:16]>> [music]