From Blind Spots to Merged PRs: Continuous Agentic Performance Optimization - May Walter, Hud

[00:00:01]Hi everyone.

[00:00:02]I don't know if you're familiar with what I'm about to show.

[00:00:05]But remember when someone from the product is like saying that some page is too slow and maybe we can optimize it.

[00:00:12]And then someone from engineering would say, "Yeah, probably what we'd have to dig in to find out." So, we just kind of leave it as is.

[00:00:21]And then a few weeks later, Jenny would say, "No, no, no, but it's actually way too slow now. We have to prioritize it. How long is it going to take?"

[00:00:30]And we would say something like, "Well, somewhere between an hour and a week.

[00:00:34]We'd have to look at it to find out."

[00:00:38]And then when she asked who can even take it, the answer is only Dave. He's the only one who kind of knows that code and everyone else who wrote it left a decade ago.

[00:00:50]And amazingly, this still happens in teams all the time because we would know how much time it takes to do a specific optimization, but we never know how long it's going to take to investigate it and to find it. And miraculously, every time we look, we we actually find things that can be done around performance and stability, but we never stop to proactively look for them because it doesn't make sense.

[00:01:16]So, hi. I'm Mike, co-founder and CTO at Thundra. We're building a runtime intelligence layer for coding agents that captures functional level context and deep forensic context on things that matter so that coding agents can help you fix what's going on with production.

[00:01:31]And as a part of that, we built an agentic workflow that helps with continuously optimizing performance of production applications.

[00:01:39]And I'm going to share a bit more about what we built, what were the challenges along the way, and hopefully you can take something out of it and apply it in your day-to-day.

[00:01:50]So, we're surrounded by a lot of agentic PRs that all look kind of good to us.

[00:01:57]And one agent wrote them and the other one says, "Yeah, I I went over it and it looks fine." And we still feel that urge to to verify before we just merge it to production.

[00:02:12]And then when it doesn't work, we end up asking ourselves, "How the hell did we get to this place?"

[00:02:18]And if you feel like that, I want you to know that you're not alone. So, Google just published their Dora metrics for 2026 and we can see that the biggest impact of AI adoption on engineering is individual effectiveness or that feeling of, "Oh my god, I'm so fast. I can do everything in the world." And for me personally, lack of sleep is another symptom of that.

[00:02:41]But the second one is software delivery instability.

[00:02:45]And the throughput is actually not impacted as much as we expected. So, we basically feel more effective, we're more effective individually, but as a team, our throughput is kind of the same and our software breaks more often, which is not exactly what we were hoping for with this AI revolution just yet.

[00:03:04]But maybe there's an agent for that and if we can build faster with AI and we can fix faster with AI, then we can get those gains that we were talking about.

[00:03:14]So, I'm going to start with why we even wanted to do this. Then we're going to go over the tech and the process, which I think is also really important.

[00:03:22]And then share some gotchas and takeaways along the way.

[00:03:26]So, that leaks kind of faster than we can bail. We have these issues that we ignore because they're not important enough then they degrade to the point where they are important enough and then we reach this crisis mode where we are all hands-on-deck. We fix it in emergency mode and then we go straight back to ignoring, which is a leaky bucket by definition.

[00:03:49]And that mostly happens because the research phase is a black box. It could take an hour or weeks and we have to pay that debt and to make sure that we spend time and engine engineering time on it in order to even know what can be done about it. And that's hard. It's legitimately hard to prioritize something where you're not sure exactly what you're going to get out of it.

[00:04:13]But what if we can automate that investigation?

[00:04:16]So, we can basically run on a weekly basis with real production context, analyze the sweet spot in the and flag the high ROI opportunities in a way that's scored. So, it runs automatically without us having to stop and do something about it.

[00:04:31]It has the production context in mind and it can give a high ROI scored performance opportunities of the things that are easy and impactful. Kind of like that performance sprint that you run every few months, just automate it.

[00:04:46]Now, let's talk about how we can actually do it cuz the dream is very nice, but but the devil's in the details.

[00:04:54]So, first of all, we wanted an infrastructure for the agentic workflows that would be vendor neutral in terms of compute and where it runs, in terms of the harness and in also in terms of the model. Things are changing all the time.

[00:05:06]We wouldn't want to constrain ourselves to a specific vendor, specific model or anything like that.

[00:05:12]And especially like with hot, we want our customers to [clears throat] be able to use whatever agents they wish.

[00:05:19]We also wanted it to be secure in terms of the tool calls, permissions, and authentication. We want some trigger system, whether those are scheduled runs like the weekly run or a set of webhooks. For example, if we see some SLO breach, we would want to investigate it.

[00:05:34]And we really wanted it to be easy to maintain and update over time. I think one of the biggest learnings we've had with agentic workflows is that even if they work out of the box or we get to a point where we're happy about them. In time, we evolve and our expectations go up. So, just being able to maintain and update those logics and build that feedback loop was very important for us so that it's reliable and the people actually trust the outcome.

[00:06:01]Specifically, we chose to work with GitHub agent workflows for that and we can choose whatever agent we want to work with and build the workflows on top of that, but there are many other great tools that could be used for that.

[00:06:15]So, this is basically how it looks like.

[00:06:18]You can see that there's a description of the job and the goal and the analysis and then we basically go over the GitHub repository and generate that weekly deep insight report analyzing production data and finding those low-hanging high-ROI opportunities.

[00:06:36]So, for this specific setup, GitHub actions runs weekly and it uses Cloud Code, that was our specific choice, and then captures the runtime intelligence overhead via MCP so that we can basically look at the different endpoints, connect to the function level of overhead is there, and send a report to Slack. Of course, that could have been Teams or an email or anything like that. It could have been Cursor or Copilot. It's just the setup that we started with um to make sure that we have something that runs without us in the loop and sends that report to somewhere we actually live in, which is Slack.

[00:07:16]So, we want to take the production context, the traces, the queries, the latencies, and analyze them with the agent, score and flag which opportunities matter because if you can optimize something, but it runs every 3 weeks or you can like reduce 20 milliseconds, then it doesn't matter.

[00:07:36]And then the most important part here is the different the evidence. So, the agent actually fixes it reruns the tests and sees the impact that it had on that specific flow that was optimized.

[00:07:50]So that the human gets something after we already detected it. We understood the root cause.

[00:07:56]We understood why it matters to the business and we verified that the fix actually impacted that time. And then a human reviews that.

[00:08:06]And the loop is actually closed. So it's not hey, I have this idea of something that you can do. It's here's something that works.

[00:08:13]And we believe would make an impact on these specific business flows in production that are running 7,000 times a week.

[00:08:21]And then the human gets in the loop as a review gate.

[00:08:25]And of course it didn't work out of the box if you were wondering. So the first hurdle we had along the way is what we call the plausible unverified. So the agent would suggest a fix. It sounds right. It would look fairly real and then after we verified it it it just didn't work. And it's it's right that the agent suggested something that could theoretically cause that flow down. But what we wanted is to ground it on what's actually happening in production.

[00:08:57]Second part was complex queries. Um we specifically use ClickHouse. It's just an amazing columnar database but it's also slightly different than the classic SQL patterns. Now I'll talk about that in a bit. And the third one is the lazy fix.

[00:09:13]I'm sure you guys also experienced that when something throws an exception and then the agent says, "Well, maybe we can just catch that exception and then everything will be fine."

[00:09:26]But what we really want is to understand why that exception was even thrown in the first place or why are the results lagging.

[00:09:36]So those were things that we found that methodology could be very, very impactful with.

[00:09:43]It's not just the understanding of the data and how to connect it. It's also being quite thorough on what we want to do in that process and sort of building the playbook of how a senior engineer would do that.

[00:09:59]And of course, we need the context to be right.

[00:10:03]So, the problem with context there are also There are only two problems with context. You either have too much of it or you have too little of it. And what we found around runtime context, especially from production, is that more often than not, you actually have these two problems together because on one hand, you have a lot of low signal data that is hard for the agent to reason over. And on the other, you might not have all the logs and traces and metrics that you need in order to investigate that issue, which would still leave some room for assumptions and theories of that are not necessarily what our users are experiencing in production.

[00:10:44]And also, when we talk about metrics like service level, CPU and memory, or endpoints in the P90s, they are not connected to the function level. So, our coding agents reason over code and they look at these metrics and there are some relations between them, but they don't exactly speak the same language. So, when we ask questions about taking time and what can I do about it, we're often finding that there are some gaps there and again, accuracy could drop from that.

[00:11:15]So, what we did is what we call prod-to-code, which is to be able to explain what's going on in production in the same level that agents reason over because the agent's context lies on a function and file level, not on an service and endpoint level. So, basically, our context is running on a function level, and it is also connected to the endpoint or event consumer or cron job that ended up starting this task. So, you can ask a question like, "Hey, this endpoint that sometimes takes 7 seconds, where is the time spent, and what can be done about it?" Whether that's a function or an outbound call to a database, an LLM, or another microservice.

[00:11:59]And with that, you basically have the complete function level context for every single function, sort of like this service map, but on a function level of the different invocations, where they come from, how often they run, and the difference the context only when it's needed, only when we see requests that are taking longer than the P99 or some threshold that we can define. Then, we will capture those forensic evidence so that we can say, "Hey, here is an example of a request that took longer.

[00:12:29]Let's find out why."

[00:12:31]And we also have the ability to see that on top of the code, which is the HUD, the heads-up display, but but I guess it's kind of a way to explain how that data set is actually structured in a way that is much more comprehensive for a coding engineer to read and over.

[00:12:51]And then, we talked about the complex queries and what to do with them. So, the basic layer gave us the HUD query language, which are basically ClickHouse queries over that structure of functions and endpoints and forensics.

[00:13:07]On top of that, we also added a set of skills. We found that sometimes just querying the data is enough, but being able to get to the right query and to ask it again and again really created a lot of variance in our eval, and the skills actually help work with that data so that agents can use it. So for example, if we're talking about a 500, an HTTP 500, we would want to understand where that error came from.

[00:13:38]If we're talking about a memory spike, we want to understand what was running on those specific pods at that specific time where memory was higher and compare it to a baseline so that we see the diff. And all of these things were extremely helpful to be able to be a bit more methodological around how that works and to create more consistent results. And on top of that, there are a set of automations. So if we have the data, the query language, a set of skills, we can build automations on top of them like auto fixing issues as they arise or detecting dead code that isn't even running for the last 60 days and eliminating it or this automated performance improve- improvements automation that are we are talking about right now.

[00:14:31]So for performance, one example of that is to be looking for artificial delays like timeouts and sleeps and plus one queries, missing indexes, sequential assings and so on so forth.

[00:14:43]These are specific things that are much easier to find when you actually look for them. And in a code base that's 20 years old and has hundreds of contributors, it makes sense that you'll find quite a lot of those.

[00:14:59]And removing them is fairly easy and impactful.

[00:15:05]And then when you're asking something like, "Why are my endpoints taking long?"

[00:15:09]you can actually find the specific reasons and not just guess a bunch of static code analysis which could get you some result. I'm not saying it's never going to work. But, when you're talking about an automation, we have to think about how to build something that is robust enough for us to trust over time.

[00:15:31]And then we said, "Okay, now that our evas are looking good, we run weekly, we find real slow endpoints from production that are invoked. We score them, we test them, we verify them. Maybe we can just open pull requests and everyone will fix everything and the world will be amazing."

[00:15:50]Well, that's not exactly how that worked because people are still people and no one wants to wake up for, you know, a rain of 80 pull requests, as small as small as they can be. That's just not how people operate.

[00:16:06]And no one has time for that. We're too busy building other things and it's fine.

[00:16:11]So, what we actually do is we use that priority to make sure that we only flag the ones that matter and we actually started with one at a time to create that appetite and that habit. So, we look at whether this is a hot path in terms of how often it runs and how critical is it for the business.

[00:16:29]The business impact as in, you know, if this is something that has to do with payments or sign-up flows, obviously, we're more sensitive to that.

[00:16:38]Kind of like asking ourselves, would we be able to convince the product manager to prioritize it?

[00:16:43]And we also looked at the risk. If it's a risky change that requires a migration or anything like that, then obviously, it would require more time from the human that's reviewing it.

[00:16:56]And in that case, we're not necessarily looking for the highest impact ones, but for the highest ROI ones. And because we automate the investigation, we can look at the impact and the risk together and only surface and and require attention on the ones that we believe are the right ones that are worth the engineering time.

[00:17:21]So, instead of opening ADPRs, we built this human-friendly report that basically tries to convince you that it's worth your while. Something like, "Hey, this endpoint, it's usually taking around 200 ms, but every once in a in a while it takes like 45 seconds, and it happens because you're using distinct and not the search index of Mongo."

[00:17:44]And there's a very short explanation of what's happening and what's the fix. And then you can either create a ticket and fix it on your own, and create a PR, or just look at it. And we found that building these small gists that are humanly readable and it's easy to understand what's the thing, made a huge, huge difference.

[00:18:09]Because we're still living in this hybrid world where humans are in the loop, and we want to respect our place in people's lives and to make sure that we flag things that really matter, and that we have some confirmation, not only that the fix is good, but also that this issue is worth fixing.

[00:18:29]And then, when you look at that endpoint and you deploy that change, and all of a sudden it's flat again, it is pretty satisfying. And then next time you'll get that report, maybe you'll have a look, and it will be easier to convince you.

[00:18:46]So, four things that I learned that could be relevant for you.

[00:18:50]One is we need to define what matters, and the scoring and the guardrails are what makes this reliable. We can automate a bunch of things, and it becomes easier to just create some slop, but when we start with what are the things that are worth it, even though it's much cheaper to fix these things than it was a few years ago, it's still not free and therefore we need to understand that it's worth the impact the human reviewer and the risk that it entails if it does.

[00:19:22]Second part is that a lot of what we're talking about today is accelerating developers and what they do and I think what's interesting about this experience is that we automated something that it's not just doing it faster.

[00:19:37]We're automating a phase that just did not happen in the day-to-day lives of engineers. No one actually stopped every week and had a look on whether there are low-hanging fruits that could be relevant. But now we can use the agents to not only do the things that we're doing faster, but also to help us surface opportunities that we would probably never do without them.

[00:20:02]This is It sounds simple, but it's hard to actually apply it in the day-to-day, but context over cleverness works almost every time. If they have the right context, if they have the right skills, if they know exactly what these agents get much much more useful and every time a new model comes out, things get better and yet I do believe that at least for most of the cases, the models are good enough already to be able to to automate that.

[00:20:29]But it is up to us to help steer and guide to the right directions to make sure that we get the results that we want. They're not necessarily the the absolute right thing because it doesn't necessarily exist and you know, when we know our domain and our business, we understand which issues matter more.

[00:20:50]What's worth fixing, what latencies actually impact our customers' experience in the most significant way.

[00:20:58]And that helps us make sure that we focus on the right things.

[00:21:02]And in that aspect, agentic engineering is not like coding with an agent.

[00:21:08]If something works 80% of the time and you're using it with your cursor in your IDE, that's fine because you're there, you're in context, and you can help fix and steer.

[00:21:19]If we're talking about an automation that runs autonomously, we have to have a very high confidence that we're doing the right thing and that it's not going to just stray off and hand us a bunch of things that we would either waste time on reviewing or just not be able to trust over time. And I think like the hardest part of this automation was to get to a point where we feel confident enough that the issue is worth fixing and that the fix is verified in runtime so that it is handed to a person when we have a fairly good confidence that it actually works.

[00:22:03]And yet, they still review it, but we know it's worth their time. And getting to that agentic engineering automation level requires crossing towards the 80-90% trust, and it's something that is dramatically different than using an agent directly as an engineer.

[00:22:26]So, I hope that was helpful.

[00:22:29]And if there are any questions or anything like that, I'm available and always love to geek out on the AI and the LDC, and uh looking forward to hear about cool automations that you build on your own.

[00:22:42]Thank you.