Prompt caching: stop wasting your tokens!

[00:00:00]Y'all, this video. This video is on prompt caching and this is the hardest video I think I've ever made. I think I made this video eight times because every time I make it, the demos fail or something goes wrong or I say something that just ends up being categorically incorrect. So, I think I've got it though and in this video I want to dive into prompt caching and show you how it actually works, what the harness does for you and what you need to just be aware of so that you don't cause a cast cash cast cash miss or cash break cuz that is going to save you a bunch of tokens. So, if you're ready and I'm not 100% sure that I am.

[00:00:39]Let's do it. All right, we are starting out in Visual Studio Code and because there are so many nuances and edge cases to prompt caching, we are just going to talk about OpenAI models today.

[00:00:49]Although, I'll try to point out differences with Anthropic as we go. So, just to start off, I'm going to send hello world here and I'm using GPT-56 terabyte and I've got an extension called the cast drop notifier. I'll put a link to that in the video description.

[00:01:05]You can install in GitHub Copilot and this just tells us when the cash breaks or when tokens were actually reused. So, you can see here zero tokens are reused and that makes sense because it's the very first prompt. There's nothing to reuse and also if you look in the corner it says 6.64 AI credits. Well, one AI credit is roughly 1 cent for those of you who do not use GitHub Copilot. So, this was about 6.64 cents to send this prompt. If we send a hello world again, the exact same prompt you'll notice that we go from 6.64 to 7.21.

[00:01:41]So, we're going up marginally compared to the first time when we went up by six, the second time we went up by less than one AI credit and that's because as you can see, 21,227 tokens were reused on GPT-56 terabyte and that friends is prompt caching in action.

[00:02:00]Now, let's talk for a second about what prompt caching actually is. So, when you send your message, you're typing hello, and then that hello gets added to what we call the system prompt, and Copilot builds this request for you in the background. The system prompt contains instructions, tools and MCP definitions, skills, custom agents, and the conversation history. Your whole chat thread is just this one really long string that just keeps getting appended to. It's quite simple. Now, this whole thing, this request then goes to the model, and the model then has to get it into a format that the model can understand, because models don't speak English, they only speak math. And so, the first thing that happens is this process called tokenization. And tokenization is where we take parts of words, or whole words, or punctuation, or space, it just depends on what tokenizer we're using. And we convert those into arbitrary numbers. They're basically IDs that represent those words.

[00:02:59]And then, tokens get turned into vectors, and vectors are like really long arrays. And these vectors are basically, essentially, they're pointing to coordinates in 600-dimensional space where this token is. Okay? And then, this third step here, compute attention, is the most important. And this is where it takes these vectors, and for each vector, right, that's lined up in the sentence, the vector represent sending the word, or parts of a word, or a space, each vector looks backwards at the previous vectors, and then the model adjusts the numbers in that vector, and the model is essentially moving these vectors around in 600-dimensional space to bring them close together so that it can mathematically represent the meaning of the word. That is just incredible.

[00:03:51]Like, the technology behind LLMs, and how these things are actually working, truly remarkable. I didn't do it justice, but now you basically understand what's going on. This computer tension step is very expensive.

[00:04:03]And so what happens is the provider, OpenAI or Anthropic, caches everything that you send to them for a period of time. The first time we send hello world, it's getting the system prompt and our hello world. That gets tokenized, turned into vectors, tension gets computed, and then all of that gets cached.

[00:04:24]And we call this the prefix. So the second time we send hello world, only that new hello world has changed. The prefix, which is everything that we sent before, is the same.

[00:04:36]So now all we have to do is read that from cache, and they'll only compute attention for the new hello world. And that is essentially how prompt caching works. However, if we change something in this system prompt, imagine we add some instructions, custom instructions, tools, and CP server skills, that is going to change what we call the prefix.

[00:04:59]And then this cache isn't going to be any good anymore because it's changed, so it's going to have to recompute it all over again, and we'll pay full price for that. Now you should note that when you read from the prompt cache, it isn't free here. It's super cheap. It's like a 90% discount, but you're still paying a small amount. You also pay a small amount to write to the cache, but either way, caching is just exponentially cheaper than having to compute the attention every single time. Now, one other thing that we're going to look at here before we look at prompt caching and what actually breaks it is that it's important to note that there's not just one thing in the cache. You have the initial cache here, and then if you make a change that changes the prefix. So let's say you disable or add an MCP server, that's going to cause a cache miss, but then that prefix also gets stored. So, you can think of these like it's like a graph or like diffs. So, OpenAI is storing the initial prompt you sent, variation one, variation two, variation three. So, as you're changing things, if you go back to states that you were in before, the cache is still there if and only if you're still within the time to live. So, that's the first tip here is that in the cache for OpenAI, you have about 30 minutes with 5 6 models for the time to live on the prompt cache. It's not been 30 minutes for us yet, so the cache is likely still here and it is.

[00:06:25]Outside of that window, the cache is going to be expired. It's no longer there. You're going to pay full price for the next prompt that you sent. For Anthropic, it's 5 minutes on the API.

[00:06:36]Now, this can be adjusted to an hour, but depending on the harness you use, that may or may not be the case because this is more expensive. Although, I believe it is an hour in Claude code.

[00:06:46]Okay, let's take a look at things that do cause the cache to break or miss. And we don't say bust because the cache is still there most likely. It's just that we missed it. We say that we missed it.

[00:06:56]So, let's take a look at that.

[00:06:58]If we were to change the model, that is by definition going to cause a miss on the cache because the model providers have the cache, not GitHub Copilot, not your harness. And so, if you switch from one provider to the other, they don't share caches, you're going to get a cache break. But, it doesn't matter even if you switch models within the same family. So, let's go to Luna here. You're likely going to get a cache break here because you changed models even within the same family. So, you can see your cache zero tokens got reused right there because the cache does not work across models. But, this is even true if we change this again. Let's go Hello World.

[00:07:41]Now, we're using 21 for 11 cash tokens, 21 506, beautiful.

[00:07:46]If we go back and even just change the reasoning level, so let's go from medium to low here, and let's send this again.

[00:07:53]And my experience, even just changing the reasoning level, as you see, causes a cash break. Right? So, if we send, let's prime this a little bit more, send hello world. Here we go, building up our cash again, beautiful. Now, if we switch back, remember how we talked about how the prompt cash, there isn't just one, but there's it keeps multiple variants as it goes in kind of like a graph.

[00:08:15]Because we have seen this medium reasoning level before, we've already done this once, going to a different reasoning level is like going to a different model, but even when you switch models and switch back, the cash is still there.

[00:08:29]Right? Now, that's if you're still within the time to live. So, if it's been 30 minutes since you switched back.

[00:08:35]So, just keep in mind that your cash may or may not still be there, and also keep in mind that I believe there is a limit to how many of these variations it keeps. So, if you do a lot of changes, things may or may not be there as you go back and forth.

[00:08:49]So, then let's take a look now at skills.

[00:08:52]For skills, I'm going to go ahead and enable this skill here, and then let's go ahead and send hello world, and we should see a cash break here, and we do.

[00:09:01]And that's because we changed the prefix. Now, as we've already talked about, because uh we have variations of the prompt, when you go back and enable this again, you would expect that there will not be a cash break, and there isn't. Right?

[00:09:15]Now, the same is basically true for MCP servers. If you add an MCP server, or if you enable one or disable one, you'll get the exact same behavior here, where you will cause a cash break. And if you go back to one that was previously enabled that you disabled, then the cash is probably still there, and that will work.

[00:09:36]Now, let's talk about custom instructions. I want to add a custom instructions file here, so let's add a new file and let's call it agents agents.indeed.

[00:09:46]And then in here, what I want to do is just when the user says hello world, respond with some random ASCII art.

[00:09:55]Okay, and then we'll save our instructions file and we would expect that this will cause a cache break. So, let's say hello world here and what happens?

[00:10:07]Huh, no cache break. But, also notice that it didn't seem to pay any attention to the instructions that we gave it. And this is an important point to know is that when you have an agents.indeed file and you if you just add it in or you make changes to that file, they do not get picked up until you start a new session with GitHub Copilot, that's with clear or new. So, let me go ahead and do hello world again and this time we should get some random ASCII. There we do, very nice. Uh let's see what else we get. I was getting some creepy stuff a V, okay.

[00:10:36]No idea why.

[00:10:38]What do we got? What is that? This is what I'm talking about. All right, the last thing that I want to show you is these custom agents. So, if we go to an agent here, let's go ahead and pick Anvil. And by the way, Anvil is my custom agent, I created it. I would highly recommend that you stop using custom agents. They're very token intensive. Instead, use skills and instructions. Uh don't try to override the behavior of the harness. So, let's go ahead and send a hello world here and I would expect here that we will see a cache break and we do. So, as you can see, there's a lot there. It's nuanced and we we only looked at a tiny portion of it, right? If we throw in Anthropic in as another provider, this gets way more complex. Uh it works similarly but not exactly the same. I have seen instances in Anthropic where going from one model to another does not break the cache, doesn't make any sense. So, we're just not even going to go down that road in this video. Also, it's important to note that when we say there's a break or a miss, that doesn't mean it's an entire break or an entire miss because the prompt is actually cached in chunks. And those chunks, when it's looking at the cache, it's saying, "Well, did this chunk anything in this chunk change?"

[00:11:46]Well, if so, then everything after it is invalid. But we can still use the chunks before it that haven't changed. I've oversimplified a lot in this video to try to make this easier to understand, but just know that usually, even if you get a cache miss, some of that cache is still going to be good. The only exception to this is when you're outside the time to live, and then the cache is just gone. The second thing is, just remember that if you enable or disable an an MCP server or a skill, or you add one of these things or remove one, this is going to cause a cache miss. Uh but also remember that if you once you do that, if you go back within the time to live, it's still there, and you won't get the cache miss.

[00:12:30]Lastly, let me just say this. The harness is doing a ton here that we didn't even discuss in terms of setting breakpoints and tracking all this stuff to try to mitigate this for you. And prompt caching is something that the GitHub Copilot team tracks heavily. And they want to see a high high percentage on hitting the cache, and then they adjust the harness to try to save you from that. So, yes, the harness should help you, but I think it also helps to just understand what's actually happening in the background, and then how that might affect your workflows going forward. And you can install the cache break extension, which I find to be invaluable now, just super interesting to see when this actually occurs.

[00:13:13]I hope that this was helpful for you.

[00:13:15]It's been very helpful for me.

[00:13:18]Happy coding.