Why inference costs so much #substack #shorts

Added: 2026-05-17

[00:00:00]OpenAI generates 100 billion words a day. Applications like Cursor generate a billion lines of accepted code daily.

[00:00:07]Training a massive model is incredibly expensive, but it is a one-time cost.

[00:00:12]Serving that model to millions of users is an infinite recurring operational expense.

[00:00:18]To measure how efficiently hardware handles that expense, we use three core metrics. The first is time to first token or TTFT. This is the exact delay a user experiences before the initial word of a response appears, dictating whether an application feels instantly responsive or frustratingly slow.

[00:00:36]Once that first word arrives, the second metric kicks in, latency. This measures the speed at which all subsequent tokens are generated and streamed to the screen.

[00:00:45]The third metric looks at the system entirely. Throughput tracks the total volume of tokens the hardware generates concurrently across an entire batch of users. It is easy to assume that improving these three metrics simply requires buying chips with faster processing cores.