The Agent Development Lifecycle 101 by Harrison Chase

[00:00:00][music] [music] >> Hello. Hello everyone. Uh thank you for coming to this webinar. Um that was a cool intro video. Um I didn't know we were doing that. That's fun. Um uh my name's Harrison, co-founder CEO of LangChain, uh and excited to be talking about the agent development life cycle uh with everyone today. Um before we get started, few logistical things. Um there is a place uh for chat and there's a place for Q&A.

[00:00:49]Um and so if you have questions that you uh want me to answer as as we're going through the slides, I will uh answer a bunch of them at the end. So drop everything in there.

[00:01:00]Uh I will do uh the the I'll I'll walk through a presentation for the first 30 or so minutes and then answer questions for the rest of the 15.

[00:01:08]>> [gasps] >> Um this is and and this has been recorded and we will put it up on YouTube afterwards as well.

[00:01:16]Um cool. So uh let me jump into it and figure out how to share my screen.

[00:01:24]There we go. Um so yeah, want want to talk today about the agent development life cycle and and kind of like a high-level overview and introduction to it.

[00:01:34]Um we've uh we we we've basically seen that over the past year, organizations have figured out how to ship agents. And the ones that have figured out how to do that the best, um they uh they they iterate systematically. They they they ship really early and then iterate quickly. And this iteration is is done in kind of like a systematic way.

[00:01:56]Um and and the reason uh that it's important or the the the the the system that they use is is all around kind of like agent agent behavior. So, the the hardest part of building agents and shipping agents is getting them to behave reliably. And so, getting something that works, you know, initially locally or in a one-off on a Twitter demo is easy, but like shipping it reliably to 100 of users at scale, the difficulty becomes down down to like the performance and the behavior of the agent. And so, when people are iterating and when people are building these agents, that's that's a key thing that we see people focus on.

[00:02:34]And And that's what the agent development life cycle is really built around.

[00:02:39]There's these components, but all of these components come together into like how can we as developers of agents get the most out of the agents that we're building. And so, the different stages of the life cycle that we see, and then I'll I'll walk through all of these, are build, test, deploy, monitor. And then you And then when you do this for a bunch of agents, you want to you want to govern all of them as well.

[00:03:02]I will We obviously build a bunch of tools to help with all stages of this.

[00:03:11]I will try to refrain from mentioning them in in till the end. We'll We'll show where they'll fit in, but I really want this to be kind of like education and and informative. More less on kind of like our technology and more on the process and the solution that we see people getting towards.

[00:03:28]So, first, want to start with build.

[00:03:30]This is where people get started. You've got an idea for an agent, you want to build it.

[00:03:35]There's There's There's different ways to build agents.

[00:03:38]And And I think this is the I think this is a big question that you should be asking as you think about as you think about how to build the agents that you want.

[00:03:51]The the level of abstraction, the level of complexity that you want. So, there are a bunch of agent frameworks out there. This is how LangChain got started, obviously, um with the LangChain framework. The The main value prop of a lot of these is uh abstractions. Abstractions around model inputs, model outputs, tools, prompts, retrieval. Um And And this provides a standardized way for people to begin building with uh with with with agents. It makes it really easy. It has all these abstractions. Um if you're going from one project to another and it uses the same abstractions, it's easier to get on boarded. Um But But you don't always you don't always have to use a framework. Um there are downsides to using a framework. Um the downsides uh might be that they they obfuscate some of what happens under the hood. Um This is a lesson we learned early on with LangChain. Um and and uh we we we addressed this by making the framework kind of like less heavy and less high-level. Um but I I remember early on uh we we had this chain and this chain made five LLM calls under the hood. And uh someone uh someone asked, "Hey, like why is this chain taking five times as long as when I call OpenAI directly?" And the reason it was doing those five The reason was it was doing those five calls under the hood, but it just wasn't obvious. It would the people who the person who was using it wasn't aware of that. And And that's um you know part That's partially better docs.

[00:05:15]That's also partially the framework's fault. It doesn't It doesn't kind of like sometimes make that clear. We spent a lot of time to make LangChain more lightweight and and and less um obfuscating of those things, but there are pros and cons to to to using frameworks.

[00:05:28]Um runtimes are another version of this.

[00:05:30]So, So, runtimes are a little bit more aimed at some of the uh runtime considerations. So, uh how do you do state management? How do you uh do uh durable execution? How do you have human intervention?

[00:05:43]These are framework uh agnostic concerns, um but they have to do with the runtime of of the agents and how they run.

[00:05:52]Um, and then harnesses are a little bit higher level than frameworks in my mind.

[00:05:56]Harnesses, uh, have been, uh, a really hot topic recently, largely because a lot of the coding agents that are out there are basically harnesses. Um, they, uh, they they are coding specific harnesses, uh, but there are a lot of learnings from those coding specific harnesses that you can port back into general purpose harnesses, um, such as, uh, state management, um, access to a file system as a way to kind of like manage context. So, context management as well, summary and compaction, um, uh, a a really clean human in the loop interface, sub agent support, skills, all of these are things that are a little bit higher level than a framework in my opinion, um, and and and part of the harness. Um, just mapping these really quickly onto the LangChain ecosystem. Actually, I'll map map it onto the whole open source ecosystem. So, frameworks, you've got things like LangChain, Llama Index, Crew AI, Google ADK, OpenAI Agents SDK, AWS Swarm, things like that. Runtimes, uh, we consider so so LangGraph, we consider that part a runtime and part a framework, um, but LangGraph is a runtime, Temporal is a runtime, um, and then harnesses, this is where you have Deep Agents, you have Claude Agents SDK, uh, those are really like the main, um, harnesses that are out there. There are coding agent harnesses, so you could also consider CodeX or Open Code, um, or Pi. Uh, Pi Pi is maybe a good example of a developer harness as well. Um, but there there's a really thin line in my mind between kind of like a general purpose developer harness and then like a coding agent harness, um, and they look very similar a bunch.

[00:07:33]Um, and and we have stuff on all our documentations about the the differences between these and the differences between the the LangChain offerings and other ones out there.

[00:07:42]No code I want to explicitly call out.

[00:07:44]This is really interesting to us.

[00:07:48]How how far can you go in in building agents if if you don't write code? And I think like one of the things that we've seen is that a lot of the agent definitions are starting to be more less and less code like. So so what I mean by that is a key part of the agent is a prompt. A prompt is not code, it's just text. You can represent this as a markdown file as an agents.md file or something like that. Tools. Tools you can you can represent with MCP servers.

[00:08:18]Sure the MCP servers themselves are are code, but connecting to those is is is just a JSON file or an mcp.json file.

[00:08:25]Skills. Skills likewise lot of kind of like markdown files. They may contain some code in terms of like the skill scripts that that you want them to execute. But I I think there's a real argument that you can make that building agents can be done with markdown files and other files like that. And so so we I'll I'll talk about what we have in that space later on, but I think you see some efforts like Eve from Vercel to to maybe call out something that's not LangChain related as a really interesting example of representing agents on a file system and is a similar approach that we take with kind of like deep agents deploy and and fleet.

[00:09:04]So moving on to test.

[00:09:06]Test has a lot has a lot of parallels to kind of like traditional software engineering, but but it looks slightly different for agents. So so what what exactly is testing? Testing's basically running your agent against some some inputs and scoring those and then using the results to to decide how well the agent's doing.

[00:09:30]It's measuring the agent. You typically want to do this to make sure you're you're not performing badly on cases that you need to perform well on Um or that you're you're improving on on a benchmark that you want to improve on.

[00:09:44]And so these are two different two slightly different kind of like user journeys. One's more like regression testing and the other's more like benchmark hill climbing or things like that. What exactly goes into this?

[00:09:58]First, there's the inputs. What are you running the agent over? These are like expected tasks, how you expect the agent to be used. Known edge cases. You can get these from compiling them from from real traces, from dog fooding traces, from generating them synthetically, but these are the inputs.

[00:10:15]Then you you you take these inputs and you build up data sets. So so one data set may have 50 different inputs in it.

[00:10:24]These in addition to the inputs, you also have um outputs or some criteria for how to judge it. So so maybe going to this distinction a little bit. For some tasks, you can have a really clear output that you can imagine.

[00:10:41]So if you're doing a classification task, the ground truth output you can say, "Hey, this should this should be true or this should be false." And then you can compare against that.

[00:10:50]For for more open-ended things, you you don't have a ground truth, but you have a set of criteria that you want to run against. So maybe taking coding as an example, Terminal Bench 2, which is probably the leading benchmark for for coding agents.

[00:11:08]The way that it works is you get this input task, you run the coding agent, and then to score it, you don't look at the files that are changed, you actually run unit tests against them. And so this is an example of criteria-based judging.

[00:11:21]And this starts to get into metrics. And so metrics you can you can have correctness, and these can compare against the ground truth. You can you can do criteria checks, and these can run the criteria checks that are part of the of of the uh uh uh uh uh data data point.

[00:11:37]You can you can you can you can define different policies you want to be compliant with, and then and then putting all of this together, you run the model over the inputs, you score it on these metrics, and then and then this makes up an experiment. And then with a bunch of experiments, you can compare versions, catch regressions, decide readiness, things like that.

[00:11:55]Um maybe two things that I want to call out. First, uh testing both both the data set and the metrics often times needs to be application specific.

[00:12:05]Um so so thinking critically about what the inputs are, what the outputs should be, how you want to judge it, those are really hard like product questions almost.

[00:12:15]Um and so and and so there's some tooling that we can give off the shelf in LangSmith to help make this uh easier, but but you still have to you'll still have to build your own data sets, and and maybe there's some metrics you can get started with, but you'll probably have to think about defining your own.

[00:12:31]Um and then the second thing that I want to call out is is just an example of where I think the space is going here.

[00:12:37]Um so I mentioned Terminal Bench 2.

[00:12:40]Terminal Bench 2 runs on an open-source framework called Harbor. Um what Harbor does is when it runs all of these different tasks, it actually runs them all in their own sandbox.

[00:12:52]Um and so this is really useful for uh long-running stateful agents. And this is this is a different paradigm than testing kind of like LLM applications.

[00:13:01]So testing LLM applications is easier.

[00:13:03]You basically just call the LLM in a loop. Like it's it's pretty easy. Um you maybe need to worry about rate limiting, but it but it's not that difficult.

[00:13:10]Testing agents, like each agent can run for I don't know, 15, 30 minutes. It it doesn't just respond, it produces artifacts as well. It interacts with the environment. And so thinking about building these environments is really important, and Harbor is a great way to run these more long-running stateful evals. We have a bunch of integrations with Harbor that are coming out soon.

[00:13:31]Maybe going on to the next one, uh deployment. What does it look like to deploy the agent? Um so so uh deploying agents is uh obviously has some similarities to deploying software, um but there's uh differences or specific focuses that are that are really important. Um so talking a little bit about the runtime, I talked about this earlier, but things like durable execution are really important.

[00:13:53]If the agent fails halfway through a run, um can you can you just resume from that step or do you need to go back to the beginning? Um persistence, memory, like what how do you both like thread level memory as well as longer term memory that that that persists across thread. How do you how do you manage that? Um streaming, how do you stream things back from the agents? This is uh streaming obviously existed before agents, but agents have made this much more prevalent. And what do you send back? Is it is it just the tokens or is it also the thinking uh process or or um the tool calls and how do you render that? Um and then human in the loop.

[00:14:28]Human in the loop uh easy to do locally, hard to do when you're running in production at scale. How do you basically like pause the agent and then just wait for it to resume?

[00:14:36]Um other considerations, sandboxes.

[00:14:38]Sandboxes are are really top of mind right now. A lot of agents, not just coding agents, need to write and execute code.

[00:14:45]Um and how do you do that in a safe and trusted way? If it's if it's running code that you've predefined ahead of time, that's basically just calling a tool and that's that's that that's easy and pretty simple. But what if the agent is writing its own code? That's that's untrusted code. You need a safe way to do that. And so and so sandboxes are are how you will want to do this.

[00:15:03]Um virtual file systems are something that that that we think are also really interesting um for agents or just file systems in general. So agents are really good at interacting with the file systems. They know [clears throat] how to read and write contacts. They've been trained on it. They've been post trained on it. They're really really good at it.

[00:15:21]Um but sometimes spinning up a whole file system for an [clears throat] agent is a lot of overhead. And so virtual file systems are basically a fancy way of saying keep your context in a database, which is more scalable, um, and you can manage more efficiently, but expose it to the agent as a file system.

[00:15:41]Um, and and it doesn't have to be a database, it could also be S3 or anything like that. It's basically saying store the data wherever you want.

[00:15:47]Um, uh, and and then give it to the agent as a file system and let it use the the read and write and glob and grep and LS and these other tools that it already knows how to use really really well.

[00:15:58]Some interesting things here that we've done are are using like Box and Notion as virtual file systems, so you can actually use, uh, human, um, uh, you know, like common human, uh, scratchpads as as virtual file systems for agents. Um, we also built, uh, our own. This is this is prompt and context hub. So prompt a little bit unrelated to this, but this exists before agents, but basically storing version prompts.

[00:16:22]Context hub is basically a way to store context. Context is just files. You can then pretty easily mount this context hub, um, as, uh, uh, uh, as a virtual file system for for agents. Um, and so you can store version prompts and skills and instructions there.

[00:16:41]Going on to to to monitoring, um, as as the fourth thing here, um, monitoring is really about showing you what, uh, what what your agent did. Um, and there's different there's different levels of this. So first up is just tracing and you can see it you can see an example of this on the right. And this is the most absolutely basic thing that like honestly you should probably have set up even before you go to production.

[00:17:05]Um, it is is just tracing of your agents.

[00:17:07]What are what are all the steps that it took? What are the inputs to each of those steps? What are the outputs of those steps? Um, seeing seeing the different tool calls that are made, the sub-agent tool calls, be able to understand exactly what the input to an LLM is, and then exactly what the output is, super important and necessary for debugging uh these LLMs and and these agents. And again, like most of the time when these agents mess up, they mess up because um the LLM messes up and then messes up because it doesn't have the right context. And so being able to see the context it has probably the most important thing that that that that you can do in this. And that's and and and so that's tracing.

[00:17:45]Um but you also uh want to start collecting feedback on these traces.

[00:17:50]Because as you're running this agent in production, you might get thousands or millions of traces. How do you know which ones to look at?

[00:17:56]Um and so this is where uh signals like uh tracking uh user thumbs up um or or or uh any feedback that the user gives is really important. This is also where online evals comes in handy.

[00:18:09]So what online evals means is basically you've got these production traces, can you run some evaluators over uh those traces and uh a and um and score them on some dimensions and then attach that as feedback. Um and so you know, this is a little bit different from evals previously because you don't know what the ground truth should be. So you have to run it in a in a in a in a in a way where you do it without ground truth. Um one example of this uh to make this concrete um that we are really interested in for uh for online evals is what we call like perceived error.

[00:18:46]So uh I I I would argue that when you see a transcript of a human talking to an agent, there are clues you can look for to see if the agent messed up. If if the user says, for example, "You messed up," that's a good signal that that that the agent messed up. But also if they say like, "No, you did X, you should have done Y." Or like, "You did this wrong." Or they they or you're doing coding and you see them posted put put paste back in the code snippet with an error. that that probably means that the user ran that code snippet and there was some error. And so, those are all signals of what we call like perceived error. We actually trained a small language model to detect this. And this is an example of an online eval that you can run. And then dashboards are basically aggregating all of those signals, whether it be latency, whether it be time to first token, whether it be these these these feedback or online evals and and showing you these in aggregate.

[00:19:39]>> So, those are those are obviously important part of monitoring agents as well.

[00:19:45]>> Lastly, I want to talk about govern.

[00:19:47]What what does it mean to govern agents?

[00:19:49]I think there's a few different aspects of this.

[00:19:53]One that's been top of mind for us recently is cost controls.

[00:19:57]Not just for agents that we build, but agents that we use as well. We're using a ton of coding agents internally, the cost of them is getting pretty out of hand and so we want to think about how we can view costs and and and also manage costs.

[00:20:09]Tool access this is another one that we think a bunch about. When agents use tools, what tools do they have access to? Who who who are they using the tool on behalf of? If an agent uses notion, is it using it on behalf of me, on behalf of a co-worker, on behalf of a service account? What does that look like? And so, having proper controls around that is really important. And likewise around audit trails.

[00:20:30]So, what tools did did an agent call and who were they calling it on behalf of? This is where some of the observability from the previous section starts to come into play as well.

[00:20:42]And so, and so and you can see that like governance, yeah, it wraps the whole life cycle. It's not just a different step. Like tracing and evals can can can easily be involved and are big part of governance. Human in the loop, we talked about this. This is really important.

[00:20:54]Like which tools need human approval before before they're run. Discoverability of tools and of agents and then sharing of all these contexts and skills.

[00:21:07]And so, we think all of this goes in governance which wraps the whole the whole life cycle.

[00:21:15]Um One of the things that's also worth talking about is is is doing this at scale. So you know, if you're building a single agent there you you probably are figuring this out as you go along. There you you you just have to think about your single agent. You don't have to kind of like think about doing this at scale. Um When you are part of a larger organization and and want every team to be doing like these are still like emerging best practices.

[00:21:41]Um and when you want every team to be following these emerging best practices, um it's often helpful to think about how to do this best at scale. Um and so some things that we've seen be helpful here.

[00:21:51]I talked about this earlier but like um frameworks whether they're on the build side or on the evaluation side, um prevent reuse and help people onboard quickly on to new problems. So uh yeah, using frameworks in the building space is a way of standardizing how different teams build and so then you can you can you can switch between teams or onboard people. And then shared evaluation frameworks, they prevent uh every team from rebuilding basically the same tests. Again, you'll probably want some uh depending on what you're building, you you'll want some data set uh specificity um and and maybe even some some metric specificity as well. But there we do see that there are organizational level metrics that that people always want to test for on on on on every agent that goes out. And so shared evaluation frameworks [clears throat] help with that. Um you probably don't want every team thinking about how to roll their own deployment infrastructure.

[00:22:41]Infrastructure's hard and annoying. Uh and so you probably want some some central A lot of this we see end up being managed by a central platform or central core AI team.

[00:22:51]Um Tracing and feedback and dashboards. Um this is really important as well. Like building the agent is kind of the fun point part, but it's it's also Uh, it's also the fastest part of this whole cycle. Um, every everything else uh in this cycle like like operating the agent at scale, that's that's the hardest part and that's that's what takes the most as well. And so and and a key part of that is just like observing what the agent's doing and so giving teams an easy way to have that really really important and then obviously governance becomes really important at at scale as well.

[00:23:24]I do want to talk a little bit before for maybe the next five minutes before going into questions about how what we're building that LangChain fits into this. And so I already see a bunch of questions rolling into the Q&A section, but keep on keep on dropping stuff there. So how do how does everything that we work on fit into the agent development life cycle? You could you can see them all mapped out here. So in build the the three things in dark deep agents, LangChain, LangGraph, these are our open source libraries. Deep agents is an agent harness. If you're getting started building agents today, this this is what I would get started with. Deep agents builds on top of LangChain. LangChain, as I mentioned, we slimmed down the abstractions. It's a much more minimal agent agent framework.

[00:24:06]If you want to build an agent from scratch, so not an agent harness, but but but start if you want to build your own harness, LangChain is is probably the best way to build your own harness.

[00:24:15]Deep agents is using our agent harness and then LangGraph is a runtime that sits beneath all of them. We've also got LangSmith Fleet, which is a no-code way to build agents. So this is this is aimed at you know, non-developers who who who want to build agents. It's entirely in the UI. A lot of it is a lot of it is is is by speaking and talking to the agent in in natural language.

[00:24:38]Um, LangSmith, so so LangSmith is our platform that that does the rest of the test, deploy, monitor, govern stuff. So we have data sets and evals and experiments in LangSmith. We have an integration with Harbor, which is the framework I mentioned earlier in LangSmith. On the deploy side, we have uh LangSmith Deployments, which uh is it it it takes your agent, uh spins up an agent server, gives it about 30 to 40 different production-ready endpoints. That includes uh streaming, uh A2A, MCP, human in the loop, short-term memory, long-term memory. It spins all that up. Um separately, we also have sandboxes. So, sandboxes, these are code sandboxes. You can use them with or without uh uh any of our open-source frameworks.

[00:25:22]That's maybe one thing I should mention as well. Um all of LangSmith is framework-agnostic. So, you can use it with us, with Vercel AI SDK, with CrewAI, with any of that. So, sandboxes are a great example of that. They're they're completely framework-agnostic.

[00:25:36]And then context hub is is basically where you can store um skills and agent.md files and use them in deep agents or just use them in your local coding agent. So, it's also it's also framework-agnostic.

[00:25:47]Monitoring, this is probably what we're best known for in LangSmith. We trace everything really well. You can um you can run online evals. We trained our own uh we trained our own uh model to detect perceived error. That's still in private beta um but we'll be rolling out uh over the next few weeks. Um we've got dashboards. You can collect user feedback. And then on the governance side, the first thing that we launched here, still in private preview, is is LLM Gateway um as a way with with cost controls and cost visibility and data privacy first and foremost in mind.

[00:26:19]Um as as we think about this whole life cycle, we we really think traces are at the center of all of this. Um so, we think again that like the hard part about building agents is getting them to behave. Um in order to get them to behave, you basically need to observe what they're doing and then tweak them and then and then measure it. And traces are really that core of of how you observe it, how you how you start to even measure it. And so, we think traces are are really the core of of all of this.

[00:26:49]Um in order to do that, uh we actually built our own database. And so it's called Smith DB.

[00:26:56]It it it is purpose-built for agent traces. My co-founder Ankush, who's much smarter than I am, leads that. And the the the reason you should care is it makes everything in LangSmith really really fast.

[00:27:07]And so storing one trace is easy.

[00:27:09]Storing billions of traces is hard.

[00:27:12]The other really cool thing that we launched recently, and I think I saw some questions in the Q&A, which I will which I'll go to after this, about like auto-improving the agent. We we want to help with that.

[00:27:25]Like you know, if you think about how humans would auto-improve an agent, they would run the agent, they would look at how it did, they would think, "Oh, it messed up on this. Let me tweak the prompt here."

[00:27:37]And then let me, you know, rerun it again and let me let me create an eval so that it so that I can measure it going forward. And then let me like redeploy it.

[00:27:47]And so we built an agent to do that. So that's LangSmith Engine. It's been out for for a month now.

[00:27:53]We think it completely changes how you think about operating these agents because it makes it so much easier.

[00:27:58]What it does is it basically runs in the background over traces. So so it sits on top of a tracing project, runs every every 6 hours, and and then basically tries to cluster traces into issues that are happening.

[00:28:14]It will it will it will create issues on this issue board. Each issue has a name, a description, and and a link to all the traces that are afflicted by this issue.

[00:28:22]But then it will also try to fix that issue. So we'll suggest code changes. It may suggest prompt changes. And then it will also do the things that you would hopefully do as as you as you fix these issues naturally yourself. It will it will try to add examples to data sets. It will create online evals to monitor this. And so this is something we're really excited about because we think it it it we think it drastically lowers the burden of operating these agents at at scale. And it's it's fun to build agents. It's it's a little bit less fun to constantly be monitoring them and playing whack-a-mole with issues. And so we hope that LangSmith engine can help with that.

[00:28:59]So with that, I'm going to go over to the QA section. There are a lot.

[00:29:05]So let's see. I'm going to I'm going to I'm going to um All right, I'm going to start all the way from the bottom. I'm going to try to do that Oh my god, there's a ton.

[00:29:20]um All right, I'm going to try to do this rapid fire.

[00:29:26]um Okay.

[00:29:28]um What do I think about Vercel's eve.dev?

[00:29:32]Yeah, this is this is a great example of representing agents as files in a file system basically.

[00:29:40]um Deep agents is already kind of similar to that. So you can point deep agents at agents.md files and scales and basically just instrument it from that.

[00:29:49]We we also have managed deep agents which takes that even a step further. um and um uh Thank you Angeline I think for putting this on screen. I didn't know how to do that. That's That's amazing.

[00:30:04]Yeah, I think I think there's still a little bit of code involved in Vercel's eve.dev.

[00:30:08]I but I think I think it's a directionally super super interesting.

[00:30:11]um to the benchmark for coding agents terminal bench two um and so yeah, that's the benchmark for coding agents that we that that most people use and it builds on top of Harbor. So Harbor's the framework that will probably be most useful for for building your own benchmarks.

[00:30:35]um What what is being done to keep the token count low inside of framework?

[00:30:40]Great question. We think about this more. So when we think of LangChain, uh we think about giving people tools to to control the token count themselves, but we don't want to have too many off-the-shelf things. We put all of those in deep agent. So deep agent is where you can see a lot of our best practices for keeping token count low.

[00:30:59]Specifically, some of the things that we do, if a tool returns a really large response, we we basically like let's say it returns 60,000 tokens. We'll we'll put that in a file. We won't show that all to the agent. And then we'll show the agent like the last thousand tokens and be like, here's the last thousand tokens.

[00:31:16]If you need more, go read from a file.

[00:31:19]And so we do that.

[00:31:21]We do summarization when it reaches some threshold.

[00:31:25]We also one one other one last clever thing we do is there's some files that have really long inputs. So if you think about writing to a file, the agent might might generate an input that is really really long.

[00:31:39]Over time that starts to add up. You can actually remove those inputs because it's already saved in the file.

[00:31:46]And so what you can do is after some period of time, you can basically remove those inputs. And if the agent really wants to remember what it wrote, it can just go read it in the file. The the thing that I'd say about all three of these is you want to do them cleverly so that you don't necessarily break prompt caching if you don't have to or you don't want to. So prompt caching is basically when if you if you send the same prompt prefix, so the like the first same number of tokens to the LLM over and over again, it will it basically like uses a cache for those.

[00:32:14]And so if you start if you start messing around with the token context that then you don't then you don't have that same amount of cache. And and and if you if you hit that cache, it's basically cheaper and faster. So you want to try to hit that cache all things else equal.

[00:32:29]So we do all these context engineering things in a way where we try to keep that cache uh, uh, relevant. What is an industry industry industry standard pass rate for LLM evaluation in percentage? This is a great question because there there there is no good answer for this. It's completely application specific. Um, it's really really hard to to give a off-the-shelf answer for this.

[00:32:52]Um, do I foresee a time where, uh, other agents will be embedded in this life cycle? For example, agents testing or monitoring other agents or will human input always be needed? Yeah, great question. LangSmith Engine is our is our answer for this. So, we very strongly believe that other agents will be involved in this life cycle.

[00:33:10]Um, we'll add more and more capabilities to Engine. At the same time, I do think human input will always will will always be needed. I think the key is figuring out scenarios where you can have an agent doing a lot of the work, but it's still presented as like a first draft that a human comes in and approves. So, this is what we do with Issue Board. Um, LangSmith Engine creates issues, but these are just issues on an issue board. If you want to apply a code change, you have to go in and press accept. It's still pretty easy, you just press one button, but you've got this human in the loop accepting it. So, so both.

[00:33:43]Um, uh, is there a way to test an agent for new {slash} novel experiences or scenarios?

[00:33:51]This is a fantastic question and one that we're thinking a lot about. Um, so, uh, um, I I I have maybe two answers here.

[00:34:02]Um, one which probably most directly involves this. Uh, you you can use uh, you can use an another agent or an LLM to generate a bunch of scenarios or inputs. The question then becomes like, how do you how do you know if the agent's responses are right or wrong there? So, one clever thing that we've done in the open source, it's called LangFuzz. It's from like two years ago. So, so we should probably update it. But, basically what you can do is you can generate two inputs to the to to to the agent that are semantically the same. They're asking the same question in just slightly different ways. And then you can run the agent over both of them.

[00:34:38]And if it gets drastically different results, you know one of them is wrong. You don't know necessarily which one, but you know one of them is wrong. And so you can then use this to identify places where it's where where it's messing up.

[00:34:53]The other thing that I want to call out here that is I just think is new around evals this concept of like environments. So as you start working with agents that are doing more and more complex things, there's a chance that they start interacting with stateful environments. So what I mean by a stateful environment is something like linear. Like if you're using linear or Jira and you're interacting with the tickets there, you may not just be reading tickets, you might be writing tickets or updating tickets.

[00:35:23]How do you do evals against this stateful environment where if you test against your linear system that changes over time and then also like you make changes on one eval run, you know, you want to reset basically to the to the to to this like clean state for the start of another eval run. So we're spending a lot of time talking and working with people on eval environments and creating these these synthetic environments to more easily test to test agents.

[00:35:55]Do you have a module to monitor the deployed agents for drift adjustment slash changes required in prompt skills references? Yeah, so so I think so two things here. Engine will do some of this. So so engine will look for issues and then and then suggest changes to prompt skills references. If you want to look more at kind of like just like trends over time, we also have a feature called insights.

[00:36:20]So insights basically does hierarchical clustering to categorize how people are using your agents. And this is where you can see drift in in topics over time.

[00:36:30]Um All right, let's go on to some more.

[00:36:34]Um Just like LangGraph Dev, is there anything for prod servers without using LangSmith uh as I have my own infrastructure and open telemetry set up? Um so, you can use uh you can use all the pieces of So, two things two answers here. One, you can use all the pieces of LangSmith um modularly. So, if you want to use just deployments and then not use observability, you totally can. If you want to use just observability and not use deployments, you also totally can.

[00:37:04]Um but then also uh we do have uh we do have some guides, although I don't think an off-the-shelf solution for for deploying LangGraph um totally open source. But, like LangGraph itself, totally open source. All the checkpointing that does the human in the loop and persistence, totally open source. And so, if you want to roll your own, you you can. We have some guides.

[00:37:25]We don't have kind of like a first-party offering that is also open source, though.

[00:37:31]Um Oh, I like this question. What do you uh when you think about agents, are you imagining AI that is performing actual work or glorified chatbot? More often not, I see businesses talking about AI agents and it's simply a chatbot with rag. Um how do we help AI agents grow beyond their definitions, do real work that removes load from humans? This is a fantastic question. Uh I totally agree that a lot of AI right now is chatbots.

[00:37:52]I absolutely think though that the future is doing work. Um I think uh I think some things that are needed. One is just like the right UI UX. If you ask AI to do work completely autonomously, the bar for doing it is going to be much higher. If you ask it to do a first draft of something, and then the human approves it, then I think that that bar uh then then I think you'll see more of these agents doing things. So, I think there's some stuff around UX to figure out. Um I also think that in terms of actually doing things I think one thing I I think like ambient style agents are really interesting here. Basically, can you run agents in the background triggered by events?

[00:38:30]They're They're seemingly always on.

[00:38:32]Um and I think this will will let humans leverage themselves more than just going to a chatbot and asking a question.

[00:38:40]Um so I so I I keep an eye out for ambient style agents here.

[00:38:46]Um um um um How do you run / set up a virtual file system? Any particular way to set it up like a rag system or other types? Like like literally a regular file system? It can be anything. Yeah, that's kind of the fun of it. It could be a regular um file system. It could be a database. Um the You can store it anywhere. The key part is just how you expose it. So we have uh in deep agents, we have uh what we call a backend interface. And so this backend interface exposes, I think, six different methods that are exposed to agents. You just implement those for whatever storage you're using and then you can use it as a virtual file system.

[00:39:22]And those six different methods are read, write, um edit, glob, grep, and ls list.

[00:39:31]What tools do you use for monitoring your agent? LangSmith all the way.

[00:39:35]Um What's the difference between testing and evals? For evals, what do you do with that user feedback? Add it to context directly. So I think So testing evals, I think generally mean the same thing. There are some distinctions like there There's There's some maybe like different terminology within this. So like I think like you could argue that like regression testing and and like benchmarking are different ways of doing testing or evals. So regression testing is more like I expect 100% kind of like pass rate.

[00:40:06]Um and and probably maybe it's testing like simpler single LLM call things.

[00:40:11]Benchmarking is like I'm at 40% right now, but I want want to hill climb to to um I don't know 90% Um how do I do that?

[00:40:22]And then there's online and offline evals.

[00:40:24]And so maybe some people mean testing to be offline evals and evals to be online evals, but yeah, testing evals I'd say same thing, but there there are different there are different kind of like terms with with within them.

[00:40:38]Um Yeah, this is another good one. How how do you evaluate agents that don't have simple success criteria? The success needs to be evaluated by an SME for example.

[00:40:52]Yeah, really good question. This this is where I think human input will always be valuable. Um We have a concept in Langsmith called annotation queues.

[00:41:05]Um that is where we see people labeling and we see humans labeling and evaluating these agent outputs. Oftentimes what they're doing is defining a criteria so that it's more scalable. So these criteria oftentimes come from SMEs, but hopefully they they are scalable and can be run without always having to have a human look at it. If you do have that always have a human look at it, we have a concept of annotation queues and and it can help with that. Um can you show us an end-to-end demo?

[00:41:36]Probably ran out of time.

[00:41:40]Although maybe okay, I'm I'm not logged into Langsmith.

[00:41:45]So no, but if you reach out we can we can have someone on our team absolutely help with that.

[00:41:53]Um Let me also stop sharing my screen.

[00:41:57]Um What are your thoughts on agent user analytics? I e analyzing what goes on between users and agents to extract business behavior insights. Do you plan to add a similar on top of LangSmith?

[00:42:18]Um, yeah, absolutely. I think uh I think this layer is really really interesting.

[00:42:23]I think online evals get at this.

[00:42:26]Um I think insights in our product get at this. I think uh a lot of times we we uh want people we want LangSmith to be used to analyze not just errors and latency, but these kind of like fuzzy uh fuzzy things.

[00:42:44]Um will online eval make agent too slow and how far can you choose your career be used to help in online eval? Yeah, this is a good question. So um so online eval runs after the agent runs. Um so it doesn't affect the agent run at all.

[00:42:57]Guardrails are very similar to online evals in that they could be testing for the same thing, but they run as the agent is running. So guardrails may make the agent slower, online evals will not make the agent slower. Online evals may cost a lot though because you're you're potentially running an LLM over all traces. That's why we trained our own small language model um because we want it to be to be cheap.

[00:43:19]Um how do you how do you think we should plan to build a self-improving agent on this framework? This is a great question. This is probably also where I will uh end off.

[00:43:29]Um I think largely because I think it this question perfectly encapsulates kind of like everything that we work on from the build to the monitor um to LangSmith engine.

[00:43:42]Um first of all like this is this is still very much like uh you know um not easy to do. Um but we are very interested in this. We put out a video today. Um it it uses LangSmith engine um as memory for agents. So specifically what we do is we take deep agents. We're running deep agents. We use a virtual file system to say, "Hey, everything in this memories folder for deep agents, this is actually stored in LangSmith context hub. We then trace everything to LangSmith and then we run engine in the background. And so engine will look at all these traces and when it sees things that the agent should remember, it will it will update the context hub. And then the deep agents, because it's connected to that context hub, it pulls it down all the time. So deep agents plus context hub plus LangSmith engine, those are three pieces of like what are what what one of our best guesses at this like self-improving agent on on this whole framework and platform looks like. We just released YouTube video for it. I think if you go to our YouTube page, it should be the the first video there, as well as a blog on Twitter as well. I'm going to go retweet it right after this. And so I yeah, this is a great way to end it. I think that encapsulates everything we do.

[00:44:52]Um with that being said, I know there are a bunch of questions here that that weren't answered. If you still have questions, feel free to reach out to me on on on on Twitter. I'll try to respond as much as possible there or or shoot us LangChain the company an email and and would be more than happy to help.

[00:45:10]Um thanks for attending. Always love the feedback, so keep it coming.

[00:45:14]Thank you guys.