Why your agents need decision traces, not just documents — Zach Blumenfeld, Neo4j

Added: 2026-05-30

[00:00:14]So, my name is Zach. Uh I work for Neo4j. We're a graph intelligence company. You can think of us like a knowledge layer graph database at the core. We help connect and resolve information so AI systems can be a little bit more accurate and explainable. Um I used to work in technical marketing. I very recently I'm going back to my AI and machine learning roots. Uh I'm transferred over to a to a research engineering role.

[00:00:40]So, I'm going to talk to you today about context graphs. How many people in this room have heard about context graphs before?

[00:00:49]Okay, so we've got about half. And how many people have actually coded anything with a context graph or with a graph like Neo4j in general?

[00:00:57]Okay. Okay, about half. So, that's good to know.

[00:01:00]So, I'm going to talk a little bit about how we think about context graphs at Neo4j, define what they are, and then I want to go over some tools um that were built by William Lyon, who's one of our product managers, um so that you can get started with them very quickly um with your own code and your agent framework.

[00:01:17]So, context graphs Foundation Capital, I think back in December.

[00:01:26]And essentially uh we saw a lot of noise around this idea uh of creating decision traces and reasoning um to help agents make better decisions.

[00:01:35]And so, to kind of think actually about what a context graph is, we need to ask ourselves, "Would you agents really be accurate?"

[00:01:43]Right? And so, um for one thing, doing a lot of retrieval, you're going to need a knowledge base, obviously. Um so, a knowledge base for something like rag or graph retrieval basically helps an agent or a chatbot answer questions correctly.

[00:02:00]What a context graph does in the evolution of this is really the information required to not only answer questions correctly, but make better decisions.

[00:02:09]So, if we took an a concrete example, and I'll show you a demo of this, financial analyst agent, um and say that this agent has a question around improving increase.

[00:02:21]And a request for a certain amount of money.

[00:02:26]To allow that agent give information about the of the system. So, customer info, transactions, and policies.

[00:02:34]And the response from that is likely to look something like this, where it could maybe assign some sort of risk score and recommend some sort of review and talk about some key risk factors. What is helpful What a context graph enables an agent to do is actually give an answer, should you reject, accept, and why, right? And it does this because in addition to getting that customer information and those transactions, it's also going to get you past decision traces and precedents, and dynamic information about why decisions are made.

[00:03:08]Um Um systems of record, really about facts, entities, current state, about precedents, causal chains, expected outcomes, and enabling the agent to act with subject matter expertise, um and telling the agent really model for a context graph.

[00:03:36]Um the entities, about things that exist, there'll be uh events, so decisions, transactions, approvals, things like that. And then context, policies that are um in different reasoning by AI that records memory, but um by employees and and past humans that have made decisions.

[00:04:03]some of it because, you know, with internet connection, I never really know, you know, how things are going to go. Um all the code for this is available. I'll I'll show links at the end. Um basically data that we generate replicates taking data in from a CRM and a support system and some other places.

[00:04:25]Um it's using um Claude in sort of the agent runtime, has some open AI embeddings. Um there's Neo4j database assisting the data with some vectors.

[00:04:36]Um and then we have our Next.js front end um as well. So, let me show you this recording running here so you can kind of see what this looks like cuz again, all these are going to go So, you asked it a question, and it will call basically a series of tools, and you can see it bringing the graph data back.

[00:05:01]Um and then you see it getting these different decision traces, and eventually it will get this reject decision, right? And if that a Then it does this thing called find precedents, which is going to be more in a second, but it's going to look at structural information in the graph to pull a bunch of And it did internet would work.

[00:06:30]connect to each other in these causal chains. So, you build off of each other.

[00:06:53]is uh So, like I was saying before, um we queried just context around Jessica's profile. We pulled back decision traces.

[00:07:03]Then we did a special of hybrid search, especially around those precedents, both on semantic similarity and structural similarity inside of the graph that actually looked at how the decision traces were made um from previous decisions, and it tried to match that, and I'll talk a little bit about how that works with graph embeddings in the next slide to ultimately come up with the recommendation.

[00:07:23]And so, we have a we have a vector index, so we're able to search like fraud rejection, for example, sort of on the semantics.

[00:07:34]But then we have this concept called graph embedding. So, you a lot of you in here are probably familiar with text embeddings, right? On on words, right? A graph embedding is the same concept except those green nodes that I was showing you before everything was connected. We actually embedded those into a vector.

[00:07:50]And so, what that means is similar decision traces are now going to be able to be looked up by vector similarity.

[00:07:57]And so, if you have system, right? Where you've had past decisions and tickets that have been Now up just like you would with vector search for for another type of text, and that might be very very hard to pull out if you just had it inside of documents.

[00:08:12]Um this is just some more information around uh the context graph demo and some of the scenarios you can run, um and some of the tools that were used here.

[00:08:22]Um GDS is called graph data science.

[00:08:24]That's what we use for the uh for the graph embeddings themselves.

[00:08:29]Um and then I'll link you over to the code here to rerun that. But what I really wanted to show you today um was another tool that was just recently created a little month ago, um which allows you to create a full um stack application to start with with just a one-line command in the terminal to actually create the context graph and the front end and the back end.

[00:08:56]Um and so, basically you can think about this like um create React app or create Next app, like you basically get this boilerplate, um and all of the scaffolding that you specify and you can specify the domain when you create the graph, um or when you create the application, rather, in the command line, um and it will give you this, you know, basically the back end, the front end, and and and everything um out of the box. So, if I was to show you uh if I go over here, kind of looks like this, and I know it's probably hard to read. There's just a UVX command, UVX create context graph, um specify the name of the app, the domain, um the framework I want to use. There's a ton of them that that we can use here. I'm choosing Pydantic AI. Um and I'm and I'm specifying demo data. And it will go and basically create this folder for me with a bunch of fixture JSON be the data in this case going to use its uh you know, basically dummy data that I can use to help host the application.

[00:09:58]And which we don't have enough time to run through all of them. It takes a few minutes to basically install the application and seed the data and stuff and start. But I do have it running. Um that it would give you I don't know how hopeful that is. It's probably hard to see.

[00:10:15]If I go back to uh right.

[00:10:25]It'll give you um you know data inside of a graph and then um you know, you can go ahead and ask questions um and it will go and and you know, basically do this type of retrieval with Cypher and and everything else. Cypher's a graph query language um to pull data back. Very similar to what we saw in that uh context graph demo that was hosted.

[00:10:53]Sorry.

[00:10:54]I'll let it run for a little bit here and um get moving cuz we only have All right. Can you hear me now?

[00:11:02]Okay. How long did that go on for by the way? How long was it?

[00:11:08]Uh just a minute.

[00:11:10]Okay.

[00:11:11]All right. Okay. Well, here's our response back. So, you can see um you know, I asked it about um prescription medications and and it and it went ahead and and pulled everything back and there's a visualization of the schema here and like I was saying different traces that you can look through.

[00:11:28]Um so, this is a great place to get started if you just want to you're interested in context graphs and you just want to code something up really quickly. Uh there's a website that we have for this too that you can go to.

[00:11:40]Um it explains this step-by-step um about how to you know, get the app started and explore everything. Um there's also some very interesting neural tools if you want to import from um software as a service. So, if you want to import from GitHub or Notion or Jira or Slack. So, you don't have to use just demo data. You can import data from these other tools.

[00:12:01]Um and then some other features and I'll give you a link to all of this at the end like I said.

[00:12:08]Um So, we were looking at a Pydantic AI application, but you can also do it with OpenAI and LangGraph and Crew and Strands and Google ADK and all these others. There's a 22 built-in domain.

[00:12:20]So, healthcare was the one that I just showed, but there's also FinServ. You can also create your own custom domain and it will actually help generate an ontology basically a graph schema for you and put everything together. We talked about um the different data connectors for GitHub, Slack, and etc. Um and it has all of the graph native stuff to power your queries um that we were seeing to pull the data, look at decision traces, um generating an MCP server and doing actually multi-turn conversation inside of that app. Um so, again, this project is uh just on it's just getting started.

[00:12:56]Um so, it's open source. People are welcome to commit um and contribute there.

[00:13:03]Now, I'll talk a little bit about what this project is uh based on. So, um underpinning this, one of the big dependencies is our Neo4j agent memory package. Um so, this is a complete memory API. Uh it includes and context graphs really need all three of these things.

[00:13:22]Um a short-term memory, a long-term memory and reasoning. So, short-term is more conversation history and session context. I think we we understand that.

[00:13:33]Uh long-term is the entities that are extracted from that um and those that repeat over time and resolving those down. And then the reasoning is going to really be sort of the um the traces inside of that context graph.

[00:13:47]Um and there's also inside of here which is very useful. I know a lot of people think about well, if I have this text data, how do I put it into a knowledge graph, right? Is is you usually one of these big questions that people have.

[00:14:00]Um so, here uh we actually implemented this inside of the package. Um where it goes through a few different stages on raw text. There's other tools that you can use sort of outside of this if you want to um take in structured data um or you can use your own language model extraction um NER processes. Um but this one just uses a few stages where you go from spaCy to gliner and and more advanced to an LLM fallback. Um and then there's a separate merging, deduplication, and enrichment strategy.

[00:14:30]So, um it's a little bit more well thought out and that helps especially with the short-term memory take that information out and transition it into useful long-term memory and entities that can be resolved over time. And the schema kind of looks like this where you have your conversations um and then in yellow you have your um your entities that get pulled out and those connect to your reasoning traces.

[00:14:55]All righty. So, we're getting towards the end here and we might actually have a few extra minutes for for questions if if that's allowed. Um but here are the resources. So, the context graph demo that I showed, if you go to that blog, it'll also link you to a live application that's just hosted. Um and so that that's very useful if you just kind of want to explore the concepts out of the box. Create context graph in the middle. On the top one, I know on our Wi-Fi for this com for some reason it blocks that website. It won't block the website if you go on a normal Wi-Fi network. I don't know why it does for this one. Um but that link does work. And then on the one on the bottom is basically just the GitHub repository.

[00:15:42]Uh Neo4j agent memory um is the agent memory package that underpins um all of that and that also integrates with Microsoft agent framework um as well as Google ADK and and a whole bunch of others.

[00:15:57]Um so, that's actually it for me. I made it through with a couple minutes to spare. So, I'm happy to open it up to any questions for the next couple minutes if if there are any in the room.

[00:16:08]Yes.

[00:16:12]Yep.

[00:16:26]Yeah, there is. You can add timestamps.

[00:16:29]Um you can add timestamps and then obviously as they occur in time the different steps, they'll be linked by like caused or next.

[00:16:37]Yeah.

[00:16:43]Um I I don't know if it does that quite yet. I think that's something that will happen as the you know, as the technology matures a little bit, but yeah, that that is a good point. It's a good idea.

[00:17:00]So, you'll prepare the ontology beforehand. For those domains that um I showed you the preexisting ones, a lot of them will use this there's something called pole-e which is like um entity um basically policy. There's like a few like predefined entity and relationship types and that's used to guide the extraction and all of the mapping and stuff. Um so, yes. Yes.

[00:17:34]Automate creating a front uh Yeah, I would look at the I would look at the create context graph package. At minimum, if you're able to describe the ontology um then describe what's in the data, um then you can definitely create a graph schema from there. Um and then that will help you do if it's unstructured do entity entity extraction.

[00:18:02]Um I it then depends on how structured your data is. So, if it's like very structured and it's in like um for example, like CSV files or tables, um then it's just a matter of like mapping the Cypher statements over.

[00:18:16]Uh yeah, so I think that that text stuff I think that can be handled inside of here. If you look at like create your custom domain, there might be some things that you can do there to help you. Again, it's a new project. So, it's all still a little bit rougher on the edges, but at least you'll see the code and you'll see how it can generate an example ontology and then how that can be fed to an extraction process to to move everything through.

[00:18:39]Yep.

[00:19:15]Yeah, I think we're still figuring that out at the moment. Um I know in in this first demo that it is like you can ask it to store like if I I for my previous conversation, I can ask it to store decisions, but I don't think it will do it unless I prompt it to.

[00:19:33]Um But yeah, that's a good point and then in the create context graph, we're still working on how you would write um new decision traces. Um so, so yeah, it's something to think about like some sort of sentiment or um like quality score on the decisions.

[00:19:52]All right.

[00:19:54]Cool. Well, thank you guys.