How LLMs Get 8x Smarter for the Exact Same Price

本站添加: 2026-05-16

[00:00:00]Let's look at the mathematics of why this is considered an absolute engineering masterpiece. You might ask, why slice the dimensions at all? Why not just run eight full-size attention mechanisms in parallel, each processing all 512 dimensions? The answer is computational cost and memory bandwidth.

[00:00:19]Matrix multiplication is incredibly expensive. It scales cubically. If we ran eight full-size attentions, the memory and computational requirements would be devastating. Running these models locally on a standard machine would melt your hardware. By dividing the Dodel dimension by the number of heads, the total amount of computation remains roughly identical to a single head attention layer. We get eight times the perspective for the exact same computational price. We define DK the dimension of the keys and queries for each head as D model divided by the number of heads. 512 / 8 gives us 64.