Tune the Harness, Before Tuning the Model with LangChain | Nemotron Labs

[00:01:02]What is up everybody? Welcome back to another Neotron Labs live stream. Uh thank you for joining us on this wonderful Tuesday. Uh we are joined today by a very special guest from Lang Chain. Uh Shri is going to be uh walking us through a little bit about what what all this harness hubhub is about. Uh you know all these uh we keep hearing about them, right? So let's uh let's learn a little bit about them. Once we're done the demo like normal, we're going to kick it off into some Q&A. So any questions you have during the demo, please put them in chat. Let us know where you're coming in from. All that kind of normal stuff. Sh, take us away.

[00:01:40]uh teach us a little bit about what uh what y'all are doing and uh what this whole harness thing is.

[00:01:46]>> Yep. Well, thank you for having me today, Chris, Sean. Excited to talk to you guys about harness engineering, Neatron 3 Ultra, some of the work we did with Nvidia. Um Amanda, could you please uh bring up my screen?

[00:02:02]Okay. Yeah. So before I kind of go into it, uh for anyone who has not worked with Langchin before, um what we essentially do is build open source frameworks and developers tooling for, you know, creating, running and evaluating LM applications and agents.

[00:02:19]And one common question I get and confusion I see is you may have heard of lang chain, LAN graph, deep agents all kind of mentioned together. So first I kind of want to separate out what those different things are before we dive into the topic at hand. So first thing is lang chain right the common one everyone has heard about uh think of lang chain as the building blocks for an agent application. So you know how you bring a model into an agent messages tools middleware all of those things. And then LAN graph is built on top of lang chain and it's used for essentially a stateful graph execution layer. So it can run the workflows, create states between steps and really helps manage that kind of end-to-end workflow. And then more so what we're seeing now and you know a big industry trend is harnesses and that's what deep agents is. Deep agents is lang chain's kind of take on an agentic harness and what it essentially does it is assembles those pieces together into a readytouse setup for you know longunning agents uh with things like you know sub aent spinning planning uh file handling all of um those sorts of things.

[00:03:40]So for I have a small demo ready for you guys. So in this case what we'll essentially be doing is we'll be working with a deep agent going over how to build it um what it consists of all those things we'll be using Neatron Ultra 3 as the model or the brain for this agent and then basically iterating about iterating through a harness and you know what to look for how do you even think about harness engineering and how do you even test these sorts of things.

[00:04:10]Um so just to start off you know making a deep agent is as simple as you know you download the package and it's creating a deep agent. In this case we set the model to again Neimatron 3 ultra.

[00:04:26]Um and yeah so as I said deep agents is a harness. So what it essentially means is a lot of the kind of capabilities and tooling you want a model to have the deep agents harness provides this out of the box.

[00:04:40]So concretely, the four kind of key capabilities we usually like to talk about is one, write to-dos for planning.

[00:04:48]So you know, as these agents are getting more complicated, they have a lot to do.

[00:04:53]Whenever a new task comes into play, you want the agent to be able to decompose those tasks into different steps, right?

[00:05:01]So say you're working on a coding project and you want it to fix a set code, right? It's not as simple as just, you know, it's saying that and it doing one little execution and it doing that, right? It has to access the file where the code lives, analyze the code, you know, maybe run some tests, figure out what's wrong, and then make the fix, test again, and you know, you have your final output. So, right there, six steps, for example, you want the agent to be able to decompose that. So, that capability comes out out of the box, and we're seeing that with a lot of harness as well. um file system capabilities. So things such as read file, grab editing file, right? Um tasks for delegation, so actually executing certain things. And then context management um for longer runs. And the context management right now uh out of the box happens a lot in uh a local kind of file system sort of way. But again, you can connect it to whatever you want um as you productionize an agent.

[00:06:04]Um, and so the default deep agent harness has to work across many different tasks and models. So there are scenarios where a specific ma model may still struggle with how one tool communicates where another model might be completely fine with how the harness is defined. So harness optimization what we're going to be talking about today essentially means that uh those failures that we're observing we want to take a look at those identify hey like in this harness why exactly is that task failing and change maybe a small subset or piece of that harness to be able to adjust for that model and then run the same test again to validate hey this change is working now able to to um what we want and to fix and find those failures without relying on vibes, right? Just like looking at something and being like, "Oh, that's not working." We need evals.

[00:07:02]Um and that's a big kind of I guess necessity as we kind of harness optimize and look into this field specifically.

[00:07:13]So, think of an eval as basically just a repeatable test. It's different from your traditional unit test because these agents are non-deterministic. We can't have your typical, you know, Python assertion test, right? So that's where an eval comes in. It has an input um the application we want to run and a scoring function that essentially looks at the result and says, hey, is this good um or is it bad? And for evals, it can be a large benchmark, but it can also be a small test created from a failure we do not want to see later.

[00:07:47]Um, and then a few practices we've been seeing with Eval specifically, start with a real task or failure, right? Um, because you know for a fact that this is how someone's engaging with your agent and if it's failing, you know for a fact that hey, we need to fix this problem.

[00:08:04]Um, and then another kind of practice we're seeing with an eval set is the idea of a golden data set, right? So before you even get a agent into production, you really want to test it to make sure that hey like is it doing everything that you want it to do? Um you know what kind of use cases do we want to cover? Uh what are some potential edge cases that might come up?

[00:08:30]And that's where the golden data set of including all those things comes in. And it's a practice that uh we've been seeing a lot.

[00:08:38]So for the purposes of this demo, what I'm going to be doing is so I have a eval um ready, right? It's I will switch. Okay, so before I go into that, so we actually have a catalog of what we call the deep agent eval set. Uh 129 eval across eight categories. And this eval set is just making sure that whenever we build out an agent with the deep agents harness, we want to ensure that it's capable of doing some you know fundamental operations in agent should we do. So just to give you guys a flavor. So file ops for example, you want it to be able to interact with your files properly, not doing some horrendous change that will just break everything later.

[00:09:26]Retrieval. Um, I'm sure you guys have heard rag systems, you know, being able to retrieve the right information, um, and append it to your kind of agents knowledge to be able to answer questions correctly. Tool use, right? So, if you have, you know, an external system you have it connected to, making sure that it can use it correctly. Yeah, just kind of um capabilities that an agent to have has to have. So, for the for this demo, I've actually picked out a um specific test. Let me see if I can find it. Um it's this test. So, test read files.

[00:10:04]I'll let you read the rest. But what essentially happens here is that in this test, we have this text file called big.tsd with 301 lines.

[00:10:16]Lines one through 300 just have the letter x. And the final line contains a unique marker called OPPOX 91. And with this test, what basically ends up happening is, you know, we'll create an agent. Um once the test is activated, that big txt file will be placed into the agent's temporary file state. Um nothing is read from my laptop. And then basically we're gonna tell the agent, hey um use, you know, read this file uh to your I I'll show you right here. So use uh read file to read yourself this txt file and tell me the exact contents of the last non-mpt line. Do not delegate this task and reply with one line. So that's the test. We have a 301 line file.

[00:11:07]Everything is X except the last one where it has um this phrase of Opo Fox 91. And so let's kick this off. So again, we have our stock agent. So what this means is just like the default deep agents harness. Um we're going to let it run. We'll see how it does.

[00:11:29]Um but you know, demos are always kind of frustrating. So I ran an instance before and so I'll switch over to show you guys how it's um working while this runs. So first of all this is Lang Smith if no one has seen this before. This is our agent engineering platform. So you know tracing uh monitoring eval data sets a bunch of different things but for now uh we'll focus on tracing.

[00:11:57]So this was a run I had done earlier and as you can see what the tracing uh really lets us see is you know what the input ask was. So this is exactly what we saw in the test all the kind of steps that the model or the agent is kind of going through. So the different middleares we're hitting um you know what model we're even calling the tools that it's accessing and whatnot. So here um some things to point out you know latency the token count but when that executes we can see that it actually did end up calling the read file um tool and if we look at the output these are the x's that I was talking about and okay it's stopping at line 99 for some reason that's interesting and then it runs again and then let's see what it outputs for us So, it's only outputting the letter X, which as we've identified before, that is not what we want to do, right? We want to see that Opal Fox 91 sort of um issue. And again, so this finished running, we see that, you know, the final answer ended up being X. Um you know, we're missing the Opal um Fox one. So again, it's you can look at these traces breaks down everything that has happened. So it's clear, right? Just to jump in for a second here, Sh right.

[00:13:24]>> Mhm. I >> mean, you're looking at the data, bro.

[00:13:27]You telling me in order to make my agents better, I have to look at data.

[00:13:30]>> Yeah, Chris, cuz think about it, right?

[00:13:32]Like, these agents are so non-deterministic. Like, we don't know what's going to happen just by looking at our code. We need to see how it performs once it executes to be able to like pinpoint the exact area of where is it failing. And this is a simple example, but as these agents get more complicated, those trace kind of trees span, you know, hundreds of levels. Um, so you really need to look at like the trace data to understand where it's failing.

[00:13:58]>> All right, fine. You you've convinced me. We still have a little data. Carry on and show us how you uh how you made it better.

[00:14:06]>> Yeah. So, as we said, the read file is stopping here for some reason. So, what is that telling me? It's telling me that for some reason when the model is calling this read file sort of tool, it's not seeing the rest of the file.

[00:14:20]Um, so there's multiple ways we can go about this, right? One is we can maybe add a prompt saying, hey, like you know this is the default limit if but keep going look at the file. But prompting is sometimes a little questionable. It might work, it might not work. So this is where I want to introduce the idea of u middleware.

[00:14:47]So what middleware essentially is is it wraps a stage of the agent loop. So it's the step between an agent steps.

[00:14:56]And so in this case we have a custom middleware called read file remaining lines middleware. So and let's go through this a little bit. Right? So first of all, I only want to activate this when um the tool I'm using is read file. So it's very specific to that. Um and then what it essentially does, right? First read file runs normally.

[00:15:25]Is this still Can you guys still see this? I just want to show the whole thing at the same time. Okay, great. So yeah, what the this middleware does is it first lets read file run normally.

[00:15:37]Then the middleware checks the return page. If that page is only part of a larger file, it adds a short note saying, "Hey, this is the total number of lines. These are how many lines are left and the next offset." So the model essentially receives the original page plus a note. So it knows that it should keep reading. And if the read already reaches the end of the file or if the tool return an error, um, the middleware kind of leaves it alone. So again it's more of like a deterministic step we can ensure will take place between uh different model calls. Um so that's a little bit about middleware and so we created that middleware and then now we'll register that new custom harness we built right so we already had the deep agents harness um but we're we want to add this middleware so we'll register it as a new harness profile as we call it. We'll add this extra middleware that we had defined above.

[00:16:35]Um so we'll create that new deep agent right because the graph state is different. And then now let's run it again. And again I had ran this before.

[00:16:47]So let's observe the trace we see for the after. So here is the upstream after. And right off the bat we can see that okay there's a lot more steps it looks like. Um, and another thing is we have read file once, twice, and three times. So if we're sticking with that 100 limit, calling three times makes sense, right? And let's see um this middleware in action and the distinction. So first of all, if we look at the after right before the read file, we see that the middleware there is the file system middleware. So the out of the box one as I mentioned before but now we have that read file um remaining lines middleware and so once we so what again what it's basically doing is you know it'll read the file and then it'll tell the model hey there's these many lines remaining um so keep going and so that's why we see this execution and then um we see the other one, third one. And then as we get to the third one, we can see that it's reading um all these lines with the 301st line finally being exposed for us to reply.

[00:18:09]And we go to the end which where we can see the output of Opal Fox 91.

[00:18:17]So right, we made a small harness change with this extra middleware um that we had defined right here. And as a result, we had um passed our kind of simple test to check that hey is the model um able to kind of do this weird file task um and do what we want. And >> boom.

[00:18:40]>> Obviously, this is one example like right when you have your golden data set or an actual agent, you want a tons tons more traces and it's going to be a more iterative loop of, you know, where is it failing? I'll fix the middleware and then I'll test again. And one practice that we did when we were harness tuning for Ultra 3 was these data sets are pretty big, right? Like as I showed you all earlier, this has 129 tests. That's really high cost that no one wants to spend constantly running on that full eval set. So when you have a big kind of set, what you want to do is you want to try to cross cluster these failures, right? Um so maybe looking at the eval set, we're seeing a consistent problem across file ops. So you know it the issue showed up in this test but not being able to read a whole file might show up in another test as well.

[00:19:38]So that's a failure pattern that we found. And then what we essentially did was you know we added this middleware in our example this read continuation or read file remaining lines middleware and we only tested on that subset of tests again to minimize cost. We want to see that, you know, it's working for like the cluster group that we're going after. And once you've done it a couple times and validated it, then only do you want to run it against the whole set.

[00:20:08]And two reasons to run against the whole set, right? One, you want to make sure that it's working across all the tests that you want it to work for. And two, sometimes what ends up happening is that certain changes like certain middleware or certain prompt changes will actually regress the scores of other test categories. So not for this um harness change, but we had one change where we were trying to address let's say this in parallel test, right? and it was working great for file operations, but when we actually ran it across all of the evals, it actually made us lose um from eight out of nine to six out of nine, in which case, you know, okay, we don't want to do that because it's hurting um some other model capabilities.

[00:20:58]>> Makes it all makes sense. Yeah. I mean, this has been fantastic demo. Let's like introduce everyone though now that we're about uh halfway through our time together. Uh if you want to Amanda just pull up that slide. Uh so uh my name is Chris. It's always Chris. Uh I'm I'm here every week. Nice to see everybody again. Max Mill MCU. See you in the chat. See a couple of others in the chat as well. Thanks for coming back. We're joined by of course Shri from Lang Chain partner engineer. Uh pretty cool. Just did a fantastic demo. And of course, we're joined by Sean Lop uh who is uh a fantastic member of our technical marketing engineering team. Uh you may know him from Nemo Claw. You may know him from basically this kind of stuff.

[00:21:47]So, uh this the idea we we have a couple questions in the in the in the chat. Uh but the first question I want to ask you Shri is a big one. Uh, Nikette Raj asks, "What does tuning the harness mean in the context of LLM agents?" So, what what the heck do we mean? We we we saw you change a middleware uh you know, is is that what harness tuning is? Like what? Help us understand.

[00:22:18]>> Yeah. So, let's take a step back, right?

[00:22:20]So, we have an agent. I'm sure you guys have heard this term being popped up so much in the past like year. And an agent comprises of the LLM, right? So, Ultra 3 in this case, and then the harness, which is essentially all the software around it. So, again, the files you give it access to, the systems you give it access to. Um, and the deep agents harness, for example, is very generic.

[00:22:45]It works across numerous models, numerous kind of tools. But some models might be, you know, better at following certain instructions um opposed to others. So in this case, what we found out is that Ultra 3 was not doing the best job with the read file continuation. So we just want to adjust that harness a little bit for Ultra 3 specifically tell it, hey, you know, beyond this 100 lines, there's more kind of uh text to read. So please do that.

[00:23:14]So it's a slight tweak in that software around the model that is more catered towards how the model operates um to be able to you know do what you bu have built it to do better.

[00:23:27]>> Yeah. So I mean the model was not good at something we could make some adjustment to our harness right that would help the model be better at that thing. that that that that that sounds simple enough, right? I mean, but it it requires you to do the things in the steps that you did, which is evaluate, understand, look at the actual data to to understand where and why, right? And and quote unquote root cause some of these issues so that you can make adjustments.

[00:23:56]>> Sean, is there anything we we missed here? Anything you want to add?

[00:24:00]>> Well, I would add, you know, why do this process? Uh a big reason is that you don't need GPUs to to do this necessarily. So you someone asked in the chat uh Tul so you asked are you guys fine-tuning the model and the the great thing about this approach is that you're actually not fine-tuning the model. So what Shrimmont was showing is harness tuning happens client side in your application code. the model, in this case Neotron Ultra, that gets to stay the same. And so what that means is for most of us who don't have a large cluster of GPUs to do model fine-tuning and then to host our fine-tuned model, we can still squeeze out this extra performance by making these changes client side. Uh, you know, another reason you might do this is because sometimes we don't always have access to the greatest smartest model or it becomes cost prohibitive to do that. And so you can use these smaller models, whether that's Neatron Ultra or Neotron Nano or you know the some of these Quen models that that are can be really effective but sometimes need some tweaks for from the harness side. Um you know a lot of times these harnesses are built kind of assuming that the first model they'll be used with is a frontier model which is great. uh but we can you kind of the whole point of this exercise is to see how much juice we can squeeze out of these more efficient models while retaining that that same performance >> and the answer is like a lot of juice right so if if you read the blog that uh that will be linked in in chat I mean it's a it's a lot uh of of juice now uh you know this idea of like nonparametric training right if we want to get real >> we want to get real fancy about it It is so useful because it lets you do it for your use case. And you know, you may have a use case that is shaped like the one you saw or the one that deep agents is served for, but this can extend across any number of use cases, you know, including your own home-cooked lang chain agents, right? That that you've you've spent time developing and and making uh perfect for your use case. We have another question, guys. Uh this one uh is a little bit we we got to answer it.

[00:26:28]Okay. From W Malik. So is the harness how it interacts with the environment or the tweak that we did? This is like a this is always a question like what the heck is a harness, right? What what what are we even talking about when we say we're tuning the harness? Like is the is the environment the harness? Is the is the is deep agents code the harness?

[00:26:46]Like what is the tree? What what is the harness? What is it?

[00:26:50]>> Yeah. So the harness like I said is essentially all the software the instructions how to execute how to work around the model guiding it towards like a specific use case or task um you want to do it it's not it's more of like so based on your question W it's basically telling the model how to interact uh with the environment as you said um so in our case like when we have file operations we saw that ultra was not reading the whole file. So the harness out of the box wasn't doing the job it needed to do. So we just added a little tweak, right? Hey, like we're going to pass the information along of, you know, once you read set amount of lines, let's tell the model how many more lines are left so it can keep reading it. So we identified where it wasn't doing the best. We added a little software nudge to give it the information it needs to be able to execute on that file operation better. And as a result, we have an agent without fine-tuning that's able to do this task um and pass it.

[00:27:55]>> Very cool. Yeah, I mean I there you go.

[00:27:57]There's a good clean answer, right? The the the idea is that you know we could by the way fine-tune guys. Uh listen, you could absolutely spend GPU hours fixing this problem. uh it is it is something that like you know you could produce a model that was better at whatever task you want. Uh but one of these options is like significantly easier to get started with and cost significantly fewer resources. And so it's good to have you know the ability to work in this space without needing to like for fork over for a for for for a training job. And if you want to learn more about that by the way hey we got some streams coming up for you. Uh, and Chris, one one thing I would throw out there also to help answer this question is the the tweak that we made that is something that can be saved in what's called a harness profile. So, so one of the things that's really cool about Langchain is that they give you this option to kind of compile all these tweaks into something that's reusable.

[00:29:01]So you know the things that Shrimoth was talking about you guys if you want to use ultra with lang chain deep agents you actually don't have to repeat that process. What we've done is taken all those tweaks and we've incorporated them into lang chain as a profile for Neatron Ultra. So anytime Deep Agents sees, hey, I'm using Neatron Ultra, it's going to apply those tweaks. And and you can do the same thing for other models, too. So save those tweaks. Uh, and that extensibility is something that's really cool and pretty unique right now to to link chain deep agents.

[00:29:37]>> And like, you know, if you fix a problem for one model, you see another h, you know, model having that problem, hey, you've already got something that uh that that that might help you, right?

[00:29:47]Like this is the uh this has been the everpresent beauty of Langchain. You do it one time and you save it forever. Uh uh Jaddenpole 4137 from YouTube asks is there a way for the models or a teacher model to do genetic evaluations of strength of two?

[00:30:06]I think maybe if I rephrase this question a little bit Jadenpole is like so you looked through data then you made some decision to to create some middleware that did something that addressed the problem then you reevaluated. That's that sure does still sound like an awful lot of me using my brain to do work, right? Is there any way for me to have uh agents or other models uh you know you know do this for us? I think that the question that we have Yeah, there's the one. Okay, this is the question we have on the screen there. Like is there any way for me to let a a model like drive this process?

[00:30:41]>> Yeah. Um that's actually a great question. Uh let me see if I can pull up an example here.

[00:30:56]So, I'm going to switch over to a um actual application that we have in production. It's called chat langchain, which essentially um helps users ask questions about uh Langchain and whatnot. And we actually have a product called engine that's built into Langmith. And what engine essentially does is that it looks through all your traces, right? So in this case, this is the failure and it had it looks through all your traces and figures out, okay, hey, it seems like in all of these scenarios, my model is not answering how I want it to answer. and you know it looks through it and actually gives us a proposed fix of okay we're seeing all these failures and I think this is what you should change in order to make this better. So yes, I think that's honestly a great approach um to kind of use a model to iterate on this harness. And to be frank, like we use the model to iterate on the harness, right? We had so many evals and so many traces to look through. It's just not possible for you to be able to do it yourself. So using any model to kind of give it those traces and tell it to analyze or using products like engine to iterate on that harness are definitely um ways to go about this. And honestly, I encourage it because you're not going to be able to look at all the traces yourself and um group all the failure modes.

[00:32:22]>> Sounds perfect, man. Maybe this is the RSI we were looking for all along, guys.

[00:32:29]>> And I think uh you Muhammad Hussein asks a pretty relevant follow-up question, which is, you know, how do you prove this process still works on on holdout cases? And I want to call out that word hold out cases as really important. You if you do the steps that Shrimmont was just describing, one thing you need to be careful about is you're overfitting to a specific failure mode and and Shrimmont talked a little bit about how you want to test things and be sure you're not accidentally regressing things and hold out cases is how you make that happen. So uh when you use a tool like Langmith Engine or any other agent to propose a fix, what you then want to do is rerun kind of your evaluation suite and understand you did the fix change the issue without introducing other problems. Uh because a lot of times especially when it comes to fixes that are around prompting the model, you can accidentally make things worse. Like a great example is you can tell the model, hey, you need to work harder. like you're stopping too soon.

[00:33:35]And that might fix one complicated eval, but it might break the like simple how do you respond to a user who says hello type of question. So there's a lot of trade-offs. Uh and using hold out cases is a good way to make sure you're respecting those trade-offs.

[00:33:52]>> We still got to use our brains, guys. We can't we can't just uh you know, unless our agent only does one thing, we can't care about only one thing. classic machine learning problem. Uh W asked a question not about deep agents but still a good one which is how do I keep track of these streams? Is there a schedule?

[00:34:10]Uh every Tuesday 11 a.m. PT. Uh you can you can join us Neotron Labs. You can follow our Nvidia AI dev channel if you want to learn more about all of our other amazing streams. uh uh a question that I have to ask uh because uh you know I I don't know if you guys remember AI in 2022 2023 right the question was always should I fine-tune or should I do prompt engineering and now it's like it's 2026 you know new year same problem should I should I fine-tune or should I do harness engineering so uh Kushagar gar asks I was trying to fine-tune model to produce better front-end pages. But now after the stream, should I focus on tuning the harness? This is like the classic. How do we approach do we is it like as easy as you start with harness tuning? You try to to get as far as you can and then if if you can't get it then we're reaching for for GPUs. Sh is is that as simple as it is or help help us think about this?

[00:35:10]>> Yeah. Um I think you hit it Chris. Like that's how I would think about it. Like Shauna said earlier, like before you go through the extensive process of figuring out GPUs, fine-tuning a model, this is a step that can easily be done to, you know, try out first to get those, in this case, front end pages to work better. But again, I would go through this process, right? I would essentially have a tracing or logs of what exactly is going on, run an agent against the logs, ask it like, hey, like what's going on? Like where are my common failures? And as a result, try to modify the harness if it's again not like I'm going back to the example, but if it's not like seeing all of the pages, for example, from a source you're using to build the front-end pages. So like a txt file or PDF to create those pages, you know, add this middleware that should fix some of it to create better front-end pages. But, you know, if you're repeatedly improving the harness and certain like failure modes are consistently happening, that's when you'd want to pivot to the fine-tuning aspect.

[00:36:16]>> It makes sense, right? Like, uh, we we want to do as much work as we can, right? As quickly as we can to make it as good as we can. And I there are going to be loads of times, if you if you read the the blog from Langchain with with Altra, you're going to see this really clear as day. loads of times we can get tons of quote quoteunquote performance as an accuracy or capability uh ceiling like that we're leaving on the table just by not doing some of these tasks that like really will will take you uh you know like a week. Uh Sean, I got a question for you. Okay, it's me online from YouTube asks the future is just big models, right? It's also small models running on PCs using uses something like Google for search and basic agent tasks.

[00:37:01]What do you think? I know Sean and I were deep in the tank last week on a on a very related problem, right, of uh AI running on your computer. Sean, what do you think? Is it is it all six trillion models from here out or or is the small model world still uh still relevant to us?

[00:37:19]>> Yeah, it's it's a great question. I mean, I think every day we're we're seeing new open models get released, which is awesome. And a lot of those models are specifically targeting these smaller footprint hardware, whether that's a a DGX Spark or a workstation or a PC. Uh, and I think what's really cool about those smaller open models combined with this idea of an open harness, what like lang chain deep agents allows you to do, is you can start to play some interesting routing games. And so what I mean by that is you can think about one aspect of harness tuning is helping the harness decide is this a question that I can answer with my small model or do I need to escalate it to a frontier model and so it's not a either or. It's really about how do we use both of these things together. Um, a concrete example of that, uh, you might be doing some really complex task, uh, like using a GPU to render a physical simulation, uh, and you might want a frontier model to help make the plan for that task, but then local models to like execute the individual steps along the way. And that type of routing that combines both things is really only possible in these open harnesses where you get to make choices like that. So, so that's why we're so excited about combining, you know, open stuff that runs on your hardware with open harnesses that let you like escalate to frontier intelligence only when you need to.

[00:38:57]>> Yeah. And uh I bet they can find a demo that looks exactly like that. Uh if if you look hard enough and pay attention to our announcements from SRAH guys, uh Sean did a lot of great work last week to uh to make this a reality. Uh Shri, I have a question, two questions. One question I'm I'm I'm interested in. The other question very straightforward technical question. So we'll start with that one. Kristoff Linsky from LinkedIn asks question uh what can we do to lower the tree recursion depths? So, uh, h, how do we fiddle with some of these deep agents, uh, you know, settings? Is there is there docs? Do we just point Claude Claude to your docs and say, "Claude, get her done, homie?" Like, what's the what's the right way to think about this?

[00:39:44]>> Yeah. Um I can't think of like the exact method but yeah there's ways that you can configure the code like you said Chris to basically reduce that tree recursion and yeah throw your cloud on it throw your codeex on it tell it to look at the docs look at your agent code and hey reduce my tree recursion there it is chat to docs guys uh if you if you want to neotron ultra I heard with deep agents uh pretty good model too so you might give that a spin uh someone from YouTube. Uh, Love Witty asks, I mean, this is a question a lot of people are going to ask, so I I'll ask it and then let us know how you feel about it. How does deep agents compare to something like Open Code? Right? So, we're saying the word harness and >> people are hearing the word harness all the time and they're hearing the word harness when they hear about OpenClaw and they're hearing the word harness when they hear about open code or pi.dev dev and they're hearing the word harness when they hear about deep uh you know uh deep agents with with like like what is these things are all called harness but like in my mind right >> they're not like the same thing I'm looking at kind of pineapple an apple and an orange and I'm being told they're all they're all fruit which is true right but like help me understand where where these things kind of sit and how we should think about you know where where lang chain fits in this in this uh space >> so just To clarify the question, Chris, um, you have open code, right? Like the open source AI coding agent. How does lang chain deep agents kind of fit into that? Is that the question?

[00:41:17]>> Yeah. Yeah. Exactly. Like how how how does how does Yeah. How does what does harness mean when we're talking about deep agents versus something like open code?

[00:41:25]>> Yeah. So, one distinction, right? So, open code is more of a coding oriented um harness. So it's going to have some instructions on its prompt on, you know, how to change files, how to test, how to work with code, things of those sorts.

[00:41:41]It might have different kind of sub aent spin-off delegation to deal with coding tasks and whatnot. Deep agents is more of a generic agentic harness. So think about like your legal ask questions agent, right? It's not really oriented towards code. But we actually have a open source harness called deep agents code that is built on top of deep agents and it has those extra kind of tooling layers to be able to uh work with coding and do those tasks. Um so yeah open code harness for coding deep agents more generic but we also have deep agents code that kind of adds the tooling needed to make it more oriented towards coding. So really encourage you all to check out deep agents code uh with ultra 3 actually. Um try it out.

[00:42:25]>> Yeah. Hell yeah. Try it out, guys. Uh, >> yeah.

[00:42:27]>> I mean, it sounds like Sean, it sounds to me like what we're saying is like harness is the big umbrella term, right?

[00:42:32]And then we have like these specific, you know, like coding harness or legal harness or, you know, whatever XYZ harness. Is this the way that you think about it like when when when you're trying trying to communicate the these topics to people?

[00:42:48]>> Yeah, that's right. And in terms of, you know, the next natural question is, you know, how do I pick which one I want?

[00:42:56]And I think there's a lot of different factors that go into that type of decision. Uh, what I usually recommend though is start by looking for examples that match for what you're trying to do.

[00:43:09]Uh, and one of the things that's awesome about Langchain is they have a ton of those examples of ways to build out using their harness these different use cases. Uh you also want to think about like you know what's the maturity of the stack you're going to build on you know how what's the team behind it uh you know who who are they working with are they going to be around uh and then also you know if you've done the work to build an evaluation set it's pretty easy to test different harnesses using the same uh evaluation set and you can you if that sounds hard you can find uh results for these different harnesses is uh across like terminal bench and other agentic benchmarks like pinchbench as well. So there's a lot of data out there to help you kind of guide uh that decision.

[00:43:59]>> There it is guys. I mean we're we're in a we're in a situation where like the words mean stuff, right? It's like harness comes with all of this baggage, right? It comes with this big idea. At the end of the day, it's like you're going to use some combination of libraries to let your model interact with something. And the techniques that you know, Shri kind of walked us through in the demo are going to be applicable regardless of what system that looks like. And you know, obviously, uh there are many great tools that exist and like Sean said, just just just measure, right? Just uh just do a little bit of science. We're we're all scientists.

[00:44:37]It's all good. we can just uh do a little bit of science, get to the bottom of the problem. Uh we're going to share some slides now. So, we're going to pay the slide tax. Thank you. Uh thank you all for for bearing with us through this. Uh if you've been hearing the word agent a lot and you're wondering what the heck is that, we've got a learning path for you. You can check it out here uh with the QR code. Basically, this is going to take you soup to nuts on uh what agents are. Heavily featuring our friends of course from Lang Chain Classic. If you go to the next slide, please. Uh we also have uh Neotron GitHub. Uh this is where we uh give you all the information you should need to run the model, run it on your your your favorite hardware, deploy it with Brev, all kinds of different things. Uh this is uh this is kind of our our Neotron uh repository. If you have any questions, raise an issue, that kind of thing.

[00:45:26]There you go. Next slide, please. Uh if you have any questions that you want answered, uh you know, especially for me, just head over to our uh Discord.

[00:45:36]I'm in there basically 247. Give me a ping. I will get to it. Uh if you have any questions lingering from the the stream, uh you know, put them in there.

[00:45:45]We'll make sure it gets to the right people to get you an answer. Uh next slide, please. If you have any questions for our community team, you can reach out directly at community.com. This is about stuff like our hackathons, our events, this kind of live stream, our other live streams, right? If you have any questions, comments, concerns or ideas, uh please put those in uh this email inbox. We have Zach who sits in a dark cave all day answering these emails. So, someone will see it. Someone will get back to you if there's something to get back to. And that's all of our slides, I believe. Look at that.

[00:46:18]Not bad. Only four slides, guys. Uh we will uh of course we have a sigraph going on right now. uh you know big big exciting kind of conference. Sean did did a really amazing kind of demo for it. Uh you know flexing some of this technology. Uh if you if you want to check that out you can find it anywhere on the web. Tons of cool people doing tons of cool stuff at that event. Uh if you uh that's it. Okay, there you go.

[00:46:46]We're done. I gotta ask this question now. So, Sean, this is the this is not the first time that you've been on the stream, right? Uh, so I'm going to ask you like what has changed since the last time we had you on the stream? Because it was like not too long ago, but it was also like long enough ago that our industry has seemingly radically changed yet again. Like, what do you what do you think your favorite thing is since like the the the Nemo Claw buzz of GTC San Jose? like what what do you think your favorite new innovation or or uh you know you hopefully you get the point.

[00:47:20]There you go.

[00:47:21]>> Yeah, I mean the thing that I'm super excited about is just how good some of these models are getting. Uh you you do need to still pair them with a powerful harness and and do that thoughtful you know scientific work we've been talking about. But, you know, 3 months ago, your optionality was kind of like, do I use codecs with 5.5 or do I use cloud with 4.8 and now it's like I can use all these different harnesses and pick from there's probably eight or nine open models that that you can choose that are g give you that same performance uh and also be quite a bit more uh cost-effective from a tokens perspective. So, it's it's really, you know, I'm I'm excited to see where the next three months take us. I'm sure everything will will change again. Uh, but yeah, the last three months, the these new open models that keep dropping are are just pretty mind-blowing.

[00:48:19]>> It has been a wild ride. If you are an enjoyer or consumer of AI, folks, that is for dang sure. Shri, I gotta ask you this question, man. uh harness tuning, harness engineering, partner engineer at Lang Chain, like what what got you here, right? Like AI is a big space, lots of stuff that needs lots of attention from all kinds of brilliant people. What what made you land in the role you're in and like what what is most exciting to you about it?

[00:48:50]>> Yeah. Um that's a interesting one. as you said like I think one thing is like the space itself the fact that it's changing constantly and honestly I feel like just the introduction of LLM has just changed just the trajectory of software the applications were able to build and all the different things we're able to do and lang chain specifically I honestly chose this place because I was I came across a lot when this whole term of agents was coming around and I think like the whole open- source aspect is pool, right? Like everyone being able to control, you know, their own model, their own harness, configure it to whatever they want to do, I think is super important. And then the partnership side can't do it ourselves, right? We don't own the whole stack. And there's so much knowledge to gain from other partnerships as well, like the work you guys are doing with Neimlaw, the work you're doing with Ultra 3. And learning from folks like yourself has also been kind of something I've always wanted to do and has appealed to me. So, it's that kind of combination of everything that has put me where I am today.

[00:50:00]>> A be truly beautiful answer, guys. Uh, we didn't even we didn't even plan that one. There you go, Shri. That was fantastic. Thank you so much for your expertise. I could not agree more. I mean, hey, we're on the Neotron Labs live stream. I could not agree more that open source is cool. Uh, I know that I believe it deeply. Sean believes it deeply and the whole Neotron uh and Nemoclaw teams behind the scenes believe it very deeply. So, thank you so much for your expertise today, guys. Thank you everyone watching, for your questions, for your comments. Uh, it's great to see everybody. We did drop Cosmos, by the way, if you want to go check it out. Thanks, Max Mill for bringing it up. Uh, that's it for us this week. We will be here next week.

[00:50:38]We're going to talk a little bit about like, okay, okay, I get it. You know, harness tuning is is not that not that bad. I can do it. It's just a little bit of science, right? Lame chain's got a bunch of tools that'll help you do it.

[00:50:48]We're going to look at next week like, hey, what if I want to actually make the models weights wiggle, right? And uh and and and do some adjustment post harness in the in the model weights. So, tune in for that if you want. We're going to show you how easy it is to customize model these days uh from a in a parametric sense. Okay. Thank you everybody for watching. We'll be here next Tuesday 11 a.m. And we'll see you then. Bye. Bye everybody.

[00:51:11]Bye everybody.