Enterprise Agents Have a Structure Problem - Ishita Daga, Tesla

[00:00:01]Hello everyone, and welcome to my talk, Enterprise Agents Have a Structural Problem.

[00:00:06]A quick introduction about myself. I'm Ishita Daga. I work as a machine learning engineer in Tesla, and I'm building enterprise agents for our organization.

[00:00:18]Before we get into the framework and the solution space, I actually want to talk about why these agents fail in the first place.

[00:00:28]So, when an agent gives a bad answer, the first reflect that we have is that we need a bigger model, we need a the latest model, we want a model with a lot of context so that it can hold a lot of information, or we need to add a lot of knowledge base in terms of dot MD files, documents, or MCP servers, plugins, etc. But, while all of these are fair solutions, they're not the answer to actually improving the data agent itself.

[00:01:00]Model The agent actually has these three main problems.

[00:01:10]One is ambiguity. So, it doesn't know what table is right, what column is right, which data source, or which knowledge base to access when, which one holds the source of truth, or like the cleanest information, which one is just a bit more flexible. So, we have a lot of these knowledge bases, but it's not defined which one is the best, um, or which one to use when.

[00:01:37]So, ambiguity, I feel, is one big problem.

[00:01:40]Second is staleness. We all know context updates so quickly. Um, a lot of decisions change, a lot of definition, KPIs, they change so often.

[00:01:51]Processes, they get updated so often.

[00:01:54]But, to keep our data agents accurate, we need to update this context, this context life cycle.

[00:02:03]And that's a second part of the problem.

[00:02:08]The last one is preference, which is more so capturing an individual or a team preference on what metric to use to answer a specific question, how to use a query, or what filters to use in a query.

[00:02:25]Although we have like a canonical query, let's say, each team has a different preference on what filters or what definitions to use. So, this I feel is very open-ended problem, still needs a lot of research, a lot of um industries or the frontier labs are actually working on solving this one.

[00:02:46]So, let's start with the first one, ambiguity.

[00:02:49]The agent needs to understand what source of truth to use, or what knowledge base to use. It can't weight all the knowledge bases equally. There has to be a bit more structure to the one it chooses to answer a particular question.

[00:03:06]In my opinion, the source of truth actually is a hierarchy, which goes from the cleanest, least flexible source of truth to the messiest, uh but most flexible, most dynamic source of truth.

[00:03:19]And the way to answer any question is to start from the cleanest to the messiest or the most dynamic one.

[00:03:26]We have divided in the sources of truth in three different buckets uh for this framework. The first one being the semantic layer, which is the best source of truth, a very curated list of all the different queries, KPI definitions, metric uh or how to calculate the metric, um the business definitions, and everything combined into just one semantic layer, which the model can use to answer a query. So, the agent just looks into the semantic layer, finds the closest KPI, and references all of these different data points from that KPI to actually give you an answer.

[00:04:08]The second part is canonical tables.

[00:04:10]These are most of the parametric tables or um queries, sorry.

[00:04:16]Where you give the agent all of a list of different parametric queries, and it sort of tries to understand which ones can it use to actually answer your question. It gives it a bit more flexibility in choosing or writing its own query or its own filters, etc. And it can answer a bit more wider set of questions.

[00:04:42]The last one is the database graph, which I feel is the most trickiest because it takes a lot of effort, but it gives you a lot of flexibility in the kind of questions that can be answered.

[00:04:54]So, the idea here is you connect each table to all the different columns uh each table to the metric that can be answered through this table and vice versa and sort of create this um enormous database graph that can be served to the agent and it can answer or query on top of it. Well, it is tricky, it is also very difficult to maintain or update. So, I feel if an enterprise is approaching or adding sources of truth, they should start adding the first and the second layer first. These are easy to set up, solve I think 80% of the problems, and then the 20% can be solved by the database graph.

[00:05:43]Moving on to our second problem, which is tailness. I think the word speaks for itself. But, the context gets rotten, or it gets deprecated, or processes change so often that it's hard to maintain dot MD files, or keep on updating your skills with the most latest context. It's just not easy, and we need a better solution for that.

[00:06:11]My answer to this problem is creating a context life cycle, which basically has two different components. The first one being embedding life data sources, or knowledge sources.

[00:06:24]By life, I mean something or the data sources that are going to be always updated, would be reviewed and well curated, and always provide the newest, or the most up-to-date data. This can include your GitHub, your um CRM tools, your Tableau, or DBT, semantic layers, whatever you want to include can be there, but it has to be a mandatory data source, like a a data source that is updated most often.

[00:06:57]The second part is a feedback loop, which is something that a lot of enterprise or data agents actually miss.

[00:07:06]The idea here is you should be able to log each and every uh event. And by event, I mean whenever someone says that this database is incorrect, or this definition definition is incorrect, or updated, or there's a new way to calculate a metric, or the there is a new filter that you need to use. All of these events need to be captured, logged, and used to update your data agent context. Once you have captured these events, the second part is to actually evaluate your data agent, and you can do that by either curating an evaluation suite with human annotated questions or human evaluations, or you can create an automated evaluation where you see all the questions that have been asked in the last couple of days and the actual answers and how close the new answers are to the actually the actual answers.

[00:08:07]So, I think there are a lot of ways you can do that, but a lot of teams do not focus on evaluation that much, and which is why the agents actually fail so often because you don't know how the agent is progressing or how is the performance. You don't keep a track of that. So, I feel creating this feedback loop or a context loop where you log, evaluate, update the context every or every so often is very important.

[00:08:37]The last problem is preference, which is a very subjective problem because preferences they change very often in terms of what kind of metric should be used, what's the right calculation.

[00:08:51]Different teams will calculate the same metric differently or um use the same query but different filters. So, there is a lot of subjectivity which needs to be captured, but it's very hard to do that on individual or team level.

[00:09:07]To give an example, um so, let's say there is team A and team B, and they don't both want to calculate the average time uh taken for a particular milestone.

[00:09:21]Team A calculates it by looking at the previous milestones or completion of the previous milestone to completion of the current milestone, while team B actually looks at the start of the current milestone to the start of the next milestone.

[00:09:34]So, while both of these are correct metrics or the correct way to calculate the metric.

[00:09:41]They both will give you very different answers and it's just about preference.

[00:09:45]We just need to understand what an individual mean when they say what's the average milestone time.

[00:09:54]So while this is a great question, I feel the industry still does not have a correct answer or correct way to solve the problem itself.

[00:10:04]I feel like there are sort of two ways to solve it. One is a semantic layer, which is something that I'm trying.

[00:10:11]Wherein you store all the different ways to calculate a metric as a semantic layer and then the individual sort of can prompt the model which one do they want to use. So you can solve it but still is not storing the preference of the user. And the second thing is you again bump into the first challenge, which is ambiguity.

[00:10:33]The agent doesn't know which metric is the right one. You still have to prompt it.

[00:10:39]Second part is agent memory, which is your mem zero or like storing memory.md file, but it still is not the best way.

[00:10:49]It stores your preference but it can't understand the distinction between two different metrics or which one to use when.

[00:10:58]So I feel there are ways to solve it but none of the ways are actually solving the preference problem itself.

[00:11:06]What we actually want is a way to route the agent to the right metric based on who what what a team or which individual is using that agent. So still an open-ended problem but yeah, I think something we should work on or research on a bit more.

[00:11:29]Which brings us to the end of this presentation. So just to conclude, I feel there are these three big problems, ambiguity, staleness, and preference.

[00:11:40]And what we really need to solve for is a better structure, a better way to manage the context and evaluate the context. And then preference, which is such an open-ended problem. It requires not just understanding how the agent works, but also embedding an individual preference, like creating a hive mind for your data agent.

[00:12:04]That's about it. Thank you so much.