The Inference Shift | Stratechery by Ben Thompson

追加: 2026-05-19

[00:00:00][music] >> The infant shift was published on Monday, May 11th, 2026. [music] If you were looking for the ideal time to IPO, being a chip company in May 2026 is hard to beat. Reuters reported over the weekend.

[00:00:14]Cerebras Systems is set to raise the size and price of its initial public offering as soon as Monday as demand for the artificial intelligence chip maker's shares continues to climb. Two people familiar with the matter told Reuters on Sunday.

[00:00:28]The company's considering a new IPO price range of 150 to 160 dollars a share, up from 115 to 125 dollars a share, and raising the number of shares marketed to 30 million from 28 million, said the sources who asked not to be identified because the information isn't public yet. The fundamental driver of the ongoing surge in semiconductor stocks is, of course, AI, particularly the realization that agents are going to need a lot of compute. What Cerebras represents, however, is something broader. While the compute story for AI has been largely GPUs, particularly from Nvidia, the future is going to look increasingly heterogeneous.

[00:01:06]The GPU era. The story of how graphics processing units became the center of AI is a well-trodden one, but in brief, just as drawing pixels on a computer screen was a parallel process, which meant there was a direct connection between the number of processing units and graphic speed, making AI-related calculations was a parallel process, which meant there was a direct connection between the number of processing units and calculation speed.

[00:01:28]Nvidia enabled this dual usage by making their graphics processors programmable and created an entire software ecosystem called CUDA to make this programming accessible.

[00:01:38]The big difference between graphics and AI has been the size of the problem being solved. Models are a lot bigger than video game textures, which has led to a dramatic expansion in high-bandwidth memory HBM per GPU, and dramatic innovations in terms of chip-to-chip networking to allow multiple chips to work together as one addressable system.

[00:01:55]Nvidia has been the leader in both.

[00:01:58]The one use case for GPUs has been training, which stretches the third point in particular. While the calculations within each training step are massively parallel, the steps themselves are serial. Every GPU has to share its results with every other GPU before the next step can begin.

[00:02:12]This is why a trillion parameter model needs to fit in the area memory of tens of thousands of GPUs that can communicate as one system.

[00:02:18]Nvidia dominates both problem spaces.

[00:02:21]First by securing HBM ahead of the rest of the industry, and second thanks to its investments in networking.

[00:02:26]Of course, training isn't the only AI workload. The other is inference.

[00:02:30]Inference has three main parts.

[00:02:32]Prefill encodes everything the LM needs to know into an understandable state.

[00:02:36]This is highly parallelizable and compute matters.

[00:02:39]The first part of decode entails reading the KV cache, which stores context including the output of the prefill step to make an attention calculation. This is a serial step where bandwidth matters, but the memory requirements are variable and increasingly large.

[00:02:52]The second part of decode is the feed forward computation over the model weights. This is also a serial step where bandwidth matters, and the memory requirements are defined by the size of the model.

[00:03:01]The two decode steps alternate for every layer of the model. They're interleaved, not in sequence, which is to say that decode is serial and memory bandwidth bound. For every token generated, two distinct memory pools must be read. The KV cache, which stores context and grows with each token, and the model weights themselves. Both must be read in full to produce a single output token.

[00:03:22]GPUs handle all three needs. High compute for prefill, abundant HBM for KV cache and model weights, and chip-to-chip networking to pull memory across multiple chips when a single GPU isn't enough.

[00:03:32]In other words, what works for training works for inference. Look no further than the deal SpaceX made with Anthropic. From Anthropic's blog, We signed an agreement with SpaceX to use all of the compute capacity at their Colossus one data center.

[00:03:46]This gives us access to more than 300 MW of new capacity, over 220,000 Nvidia GPUs within the month.

[00:03:54]This additional capacity will directly improve capacity for Claude Pro and Claude Max subscribers. SpaceX retains Colossus 2, presumably for both training of future models and inference of existing ones, and can afford to do both in the same data center precisely because xAI's models aren't getting much usage.

[00:04:12]More pertinently to this piece, they can do both in the same data center because both training and inference can be done on GPUs.

[00:04:18]Indeed, the GPUs Anthropic is contracting Colossus 1 were originally used for training as well.

[00:04:23]>> [music] >> The fact that GPUs are so flexible is a big advantage.

[00:04:28]Understanding Cerebras.

[00:04:30]Cerebras makes something completely different. While a silicon wafer has a diameter of 300 mm, the reticle limit, the maximum area that a lithography tool can expose on that wafer, is around 26 mm by 33 mm.

[00:04:42]This is the effective size limit for chips. Going beyond that entails linking two separate chips together over a chip-to-chip interposer, which is exactly what Nvidia has done with the B200. Cerebras, on the other hand, has invented a way to lay down wiring across the so-called scribe lines that are the boundary between reticle exposures, making the entire wafer into a single chip with no need for relatively slow chip-to-chip linkages.

[00:05:05]The net result is a chip with a lot of compute and a lot of SRAM that is blisteringly fast to access.

[00:05:11]To put it in numbers, the WSE3, Cerebras' latest chip, has 44 GB of on-chip SRAM at 21 TB per second of bandwidth.

[00:05:20]An H100 is 80 GB of HBM at 3.35 TB per second.

[00:05:25]In other words, the WSE3 has just over half the memory of an H100, but 6,000 times the memory bandwidth.

[00:05:33]The reason to compare the WSE3 to an H100 is that the H100 is the chip most used for inference, and inference is clearly what Cerebras is most well suited for.

[00:05:41]You can use Cerebras chips for training, but the chip-to-chip networking story isn't very compelling, which is to say that all of the compute and on-chip memory is mostly just sitting around.

[00:05:50]What is much more interesting is the idea of getting a stream of tokens at dramatically faster speed than you can from a GPU.

[00:05:56]Note, however, that the limitation in terms of training also potentially applies in terms of inference.

[00:06:01]As long as everything fits in on-chip memory, Cerebras speed is an incredible experience. The moment you need more memory, whether that be for a larger model or, more likely, a larger KV cache, then Cerebras doesn't make much sense, particularly the price. That whole wafer as chip technique means high yields are a massive challenge, which hugely drives up costs. At the same time, I do think there will be a market for Cerebras-style chips.

[00:06:26]Right now, the company is highlighting the usefulness of speed for coding.

[00:06:29]Reasoning means a lot of tokens, which means that dramatically scaling up tokens per second equals faster thinking. But, I think this is a temporary use case for reasons I'll explain in a bit.

[00:06:39]What does matter is how long humans are waiting for an answer. And as products like AI wearables become more of a thing, the speed of interaction, particularly for voice, which will be a function of token generation speed, [music] will have a tangible effect on the user experience.

[00:06:53]Agentic inference.

[00:06:55]I have previously made the case, including in Agents over Bubbles, that we have gone through three inflection points in the LLM era.

[00:07:01]ChatGPT demonstrated the utility of token prediction.

[00:07:04]01 introduced the idea of reasoning, where more tokens meant better answers.

[00:07:08]Opus 4.5 and Quad Code introduced the first usable agents, which could actually accomplish tasks using a combination of reasoning models in a harness that utilizes tools, verified work, etc. All this falls under the banner of inference, but I think it will be increasingly clear that there's a difference between providing an answer, what I will call answer inference, and doing a task, what I will call agentic inference.

[00:07:29]Cerebras's target market is answer inference. In the long run, I think the architecture for agentic inference will look a lot different, not just from Cerebras's approach, but from the GPU approach as well.

[00:07:41]I mentioned above that fast inference for coding is a temporary use case.

[00:07:44]Specifically, coding with LLMs requires a human in the loop. It's the human that defines what is to be coded, checks the work, commits the pull request, et cetera.

[00:07:52]It's not hard to envision a future, however, where all this is completely handled by machines. This will apply to agentic work broadly. The true power of agents will not be that they do work for humans, but rather that they do work without human involvement at all.

[00:08:06]This, by extension, will mean that the likely best approach for solving agentic inference will look a lot different than answer inference.

[00:08:13]The most important aspect for answer inference is token speed. The most important aspect for agentic inference, however, is memory.

[00:08:20]Agents need context, state, and history.

[00:08:22]Some of that will live as active KB cache. Some will live in host memory or SSDs. Much of it will live in databases, logs, embeddings, and object stores.

[00:08:31]The important point is that agentic inference will be less about GPUs answering a question, and more about the memory hierarchy wrapped around the model.

[00:08:39]Critically, this articulation of an agentic specific memory hierarchy implies a necessary trade-off of speed for capacity.

[00:08:45]Here's the thing, though. Lower speed isn't nearly as important a consideration if there isn't a human in the loop. If an agent is waiting around for a job that is being run overnight, the agent doesn't know or care about the user experience impact. What is most important is being able to accomplish a task, and if entirely new approaches to memory make that possible, then delays are fine.

[00:09:05]Meanwhile, if delays are fine, then all the focus on pure compute power and high bandwidth with memory seems out of place.

[00:09:12]If latency isn't the top priority, then slower and cheaper memory, like traditional DRAM, for example, makes a lot more sense.

[00:09:18]And if the entire system is mostly waiting on memory, then chips don't need to be as fast as the cutting edge, either.

[00:09:24]This represents a profound shift in future architectures, but it also doesn't mean that current architectures are going away.

[00:09:30]Training will continue to matter, and Nvidia's current architecture, including high-speed compute, large amounts of high-bandwidth memory, and high-speed networking will continue to dominate.

[00:09:39]Answer inference will be a meaningful market, albeit a relatively small one, and speed from chips like Cerebras or Groq. I explained how Nvidia is deploying Groq LPU's here will be very useful.

[00:09:50]Agentic inference will gradually unbundle the GPU, which alternates between straining high-bandwidth memory during the prefill process, and straining compute during the decode process, in favor of increasingly sophisticated memory hierarchies dominated by high-capacity and relatively lower-cost memory types with good enough compute. Indeed, if anything will be the speed of CPUs for things like tool use that will matter more than the speed of GPUs.

[00:10:15]At the same time, these categories won't be equal in size or importance.

[00:10:18]Specifically, agentic inference will be the largest market by far, because that is the market that won't be limited by humans or time.

[00:10:25]Today's agents are fancy answer inference. In the future, true agentic inference will be worked on by computers according to dictates given by other computers, and the market size scales not with humans, but with compute.

[00:10:36][music] The implications of agentic inference on compute.

[00:10:41]To date, the implication of scale with compute has implicitly meant Nvidia bullishness. However, much of Nvidia's relative advantage to date has been a function of latency.

[00:10:50]Nvidia chips have fast compute, but keeping that compute busy has required big investments in ever-expanding HBM memory and networking.

[00:10:57]If latency isn't a key constraint, however, then Nvidia's approach seems less worth paying a premium for.

[00:11:03]Nvidia does recognize this shift. The company launched an inference framework called Dynamo that helps disaggregate different parts of inference, and is shipping products like standalone memory and CPU racks to enable increasingly large KV caches and faster tool use. The better to keep their expensive GPUs busy.

[00:11:19]Ultimately, however, it's easy to see costs and simplicity being increasingly attractive to hyperscalers for agentic inference that isn't remotely GPU bound.

[00:11:28]China, meanwhile, for all of its lack of leading-edge compute, has everything it needs for agentic inference. Fast enough, but not leading-edge GPUs. Fast enough, but not leading-edge CPUs, DRAM, hard drives, et cetera.

[00:11:40]The challenge, of course, is compute for training. It's also possible that answer inference is more important for national security, at least when it comes to military applications.

[00:11:50]The other interesting angle is space.

[00:11:52]Solar chips actually make space data centers more viable for a number of reasons. First, if memory can be offloaded, chips can be made much simpler and run much cooler. Second, older nodes, by virtue of being physically larger, will better withstand space radiation. Third, older nodes require less power, which means there will be less heat to dissipate via radiation. Fourth, not being on the bleeding edge will mean higher reliability, an important consideration given that satellites won't be repairable.

[00:12:18]Nvidia CEO Jensen Huang regularly says that Moore's Law is dead. What he means is that the future of computing speed-ups will be a function of systems innovation, which is exactly what Nvidia has done.

[00:12:28]Maybe the most profound implication of agents that act without humans in the loop, however, will be that Moore's Law doesn't matter, and that the way we get more compute is by realizing that the compute we have is already good enough.