How MIT & NVIDIA Solved the Long-Context Bottleneck For AI Models #ai #llm #softwareengineering

Added: 2026-05-06

[00:00:00]If you have tried running a heavy reasoning model like DeepSeek-R 1 locally, you've probably seen your GPU memory spike really quickly. But most of that isn't because of the model weights, it's the KV cache, which is the real bottleneck. As the conversation gets longer, the memory needed to store past tokens explodes, leading to those out-of-memory errors or glacial processing speeds. Researchers from MIT and Nvidia just released a paper called Try Attention, and it's a total game-changer for long context LLMs. You see, to save memory, most models use KV cache pruning. They try to guess which tokens are important and toss the rest.

[00:00:34]The problem is that modern models use Rope or Rotary Positional Embeddings, and because tokens rotate based on their position, a query from 2 seconds ago looks completely different from one now.

[00:00:45]And trying to pick the best keys in that rotating space is like trying to catch a fish in a blender. It's unstable, and the model ends up forgetting crucial logic, which inevitably tanks its reasoning score. But the team at MIT discovered something fascinating. If you look at the vectors before the Rope rotation, the pre-Rope space, they are incredibly stable. They all cluster around fixed centers, and they realized that the attention pattern actually follows a trigonometric series. By using this math, they can predict exactly which keys a model will want to look at in the future based on their distance without needing to run a full heavy attention mechanism. And the results are wild. By using this trigonometric scoring, they've achieved a 10.7x reduction in KV cache memory, and they also boosted throughput by 2.5x. In one test, they ran a 32 billion model on a single 24 GB consumer GPU, like an RTX 3090 or a 4090, and usually that would produce an out-of-memory errors instantly with long instructions. But Try Attention compressed the cache on the fly and finished the task perfectly.

[00:01:45]For developers, this means we can finally run high-end reasoning agents on local hardware without the massive memory tax of long context windows. If you want to stay up to date with the latest efficiency breakthroughs in AI, be sure to subscribe to the Better Stack channel.