RAG vs long context: the 32K rule I use to decide

Added: 2026-05-27

[00:00:00]Every long context guide says just retrieve. Past 32,000 tokens, that advice flips. And nobody hands you the line for when to stuff it all in instead. So, I run three checks. One, does the corpus fit under 32,000 tokens?

[00:00:14]Past that, it still loads, but it skims the middle and drops things.

[00:00:18]Two is query shape. Take a 200,000 token policy dump, ask for the refund window.

[00:00:23]Stuffed. That line sits at token 140,000, [music] where attention is thinnest. Retrieval pulls the 800 token refund chunk to the top, and the model nails it. But ask it to summarize every policy change this year, [music] and retrieval is cooked. No single chunk holds that answer, so it wants the whole thing in context.

[00:00:41]Three is cost, since long context repays the full prefill on every call.

[00:00:45]>> [music] >> Sure, retrieval grabs the wrong chunk, and long context forgets its middle.

[00:00:49]Pick your failure. Size, shape, cost.

[00:00:52]The line is 32,000.