Agent Memory Explained in 5 Minutes

[00:00:00]Before agents became a major focus, most LLM applications relied almost entirely on context window. You would have a system prompt, the current user message, and maybe some recent chat history. But that was essentially all that the model could remember. But the most important thing to remember is that the model does not automatically remember everything that happened yesterday, 10 tool calls ago, or across previous sessions. the application has to put in that information back into the prompt or give the model the right tool to retrieve it.

[00:00:30]Around late 2022 and 2023 when chachet took off, memory was basically whatever we could fit into the context. And you can probably guess this creates an obvious problem. As the task got longer, agents lose track of earlier decisions, repeat mistakes, and fail to maintain coherence over time. So the first thing we need to do is to understand the difference between context and memory.

[00:00:55]Context is active. It is what the model sees during the current forward pass.

[00:01:00]But memory is different. Memory is information that is stored outside of the model that may or may not be used later on. So the key idea is that short-term and long-term memory lives in different places. A useful analogy that we might all know by is computer memory.

[00:01:16]When you're working on a document, the active version is stored in RAM. It is fast and immediately accessible, but it is not permanent. So, if your computer suddenly uses power, before you save the document, the state actually disappears.

[00:01:30]To make it more persistent, you would have to write it somewhere more durable, like a disk or a solid state drive.

[00:01:37]Agent memory works in a very similar way. The context window is kind of like RAM, where it's active, fast, and limited. And long-term memory is more like a disk persistent. But the agent needs a system for deciding what to save, what to achieve, and when to actually load it back into the context.

[00:01:55]So before agents had dedicated memory systems, the field mostly tried to work around the context window in three main ways. Rag, summaries, and profiles. Rag helped agents reference large bodies of information without stuffing everything into the prompt. Instead of putting an entire document or knowledge base into the context, the system retrieves the most relevant chunks. Summaries help to compress long conversations rather than keeping every single message. The system could just summarize everything that had happened so far. And profiles helped with personalizations. They store tables of facts about the users such as preferences, writing style, or ongoing project that the user might be working on. But each approach had its own limitations. Rag, for example, can retrieve wrong information, and it is also harder to set up than it actually looks. You need to chunk the data, embed it, index it, and retrieve the right pieces and rank them correctly. All of this is a lot of overhead to actually set it up properly. Summaries are lossy because they rely on the LLM to decide which detail are worth preserving. If the summary drops on one detail that matters later on, the agent may not be able to recover it properly. And profiles can become stale and contradictory in some cases because user behavior changes over time. Something that was true last month may no longer be true today, but the system still needs to treat it like a permanent fact.

[00:03:24]So you can imagine even when the model context gets larger, it does not fully solve the problem. And that's because memory is not just about storing more tokens, the deeper issue is memory management. And modern agents are no longer just an LLM that we ask questions. So clearly the industry needed a more serious approach to the problem when it comes to memory. As agents became more capable, memory had to become more than just an extra context. It needed its own serious layer in the agentic stack. And the reason why this layer needs to exist is because memory is not just an one-time action.

[00:04:01]It is a loop. And it looks something like this. The first half is writing memory. The agent observes what happened, decides what matters, and stores it somewhere persistent. That alone is difficult because most things that agent sees are not worth remembering forever. The second half is actually using the memory. The agent retrieves the relevant information, puts it back into the context, and updates it if something has changed. And for longunning tasks, the agent does not just need to remember the facts, it needs to remember experience as well.

[00:04:33]for example, what approach worked last time or what mistake it should avoid next time. That is why memory matters when it comes to long horizon agents benchmarks where the model has to stay coherent over many steps. And that may be one of the biggest difference between today's assistant and the next generation of agents that are coming in the future.

[00:04:54][music]