Cloudflare Replaced NGINX With Rust. Now It Runs the Cache Too

[00:00:00]Every time your browser opens a connection, there's a handshake. Two machines, a few milliseconds, agreeing to talk. Cloudflare said it was saving 434 years of that handshake time every day. That was 2022. They got there by throwing out NGX. Everybody quotes the same number, a trillion requests a day.

[00:00:19]That number is almost 4 years old. In May 2026, Cloudflare quietly gave it a second job. We'll get to what it took over, but the second job only makes sense once you know why the first machine came out. NGINX is the web server that runs a large share of the internet. To understand why you rip out working software, you have to understand how NGNX is shaped. NGNX runs a set of worker processes. A request arrives, it gets handed to one worker and that worker owns it until it's done. That design is elegant. At normal scale, it's completely fine. At Cloudflare's scale, it creates three problems. And they said all three out loud. First, a request can only ever be served by a single worker.

[00:01:00]So CPU load across cores goes lopsided.

[00:01:03]One worker gets a heavy request. The rest sit idle. Second, if a request does something CPU heavy or blocks on I/IO, it doesn't just slow itself down. It stalls every other request sitting behind it on that same worker. Third, and this is the one that actually costs money, connection pools are per worker.

[00:01:20]Every worker keeps its own pool of connections to origin servers. So the more workers you add, the worse your connection reuse gets. You scale up and your reuse ratio goes down. Pull that pool out of the workers and share it and the ratio moves. Cloudflare reported taking one customer from 87% connection reuse to 99.92.

[00:01:40]160 times fewer new connections. Same post, same year, 2022. Then there's the ceiling on customization. Cloudflare wanted things NGNX simply wouldn't do, like retrying a request against a different origin with a different set of headers. The core is C, which carries real memory safety risk. The extension path is Lua, which gives you no static typing to lean on once the logic gets complicated, and the upstream project develops behind closed doors, so the features they needed weren't going to arrive on their schedule. So, this wasn't a language preference. It was an architecture that had run out of room.

[00:02:14]The number first properly dated. In September 2022, when Cloudflare published how they built Pingora, they stated it was already serving over 1 trillion requests a day, not a month, a day. And it is the least interesting thing about this proxy. Break that down and it's roughly 11 12 million requests per second sustained as an average, not a peak. At that volume, the numbers stop looking incremental. 5 milliseconds off median time to first bite, 80 milliseconds off the 95th percentile, about 70% less CPU and 67% less memory than the service it replaced. That last pair is the one to sit with. Same traffic, a third of the machine. So why Rust and not C or go? C was already on the table. It's what NGX is written in.

[00:02:59]Cloudflare's own framing was that Rust does what C does, memory safely without giving up performance. At their scale, a memory bug isn't a bug. It's a security incident with a blast radius the size of the internet. Go is the more interesting rejection and Cloudflare didn't spell it out. Discord did publicly in 2120.

[00:03:18]Discord's red state service was fast almost all the time and then every 2 minutes it stalled a spike bad enough that users noticed. The cause was the garbage collector walking their cache to find out what was still referenced. They moved the service to Rust and the spikes disappeared because Rust frees memory when it stops being needed instead of when a collector gets around to it. A garbage collector is a scheduled interruption you don't control.

[00:03:41]Survivable in most software, not survivable in the hot path of a proxy carrying a trillion requests a day. Go collector has gotten considerably better since then. It still runs on a schedule you don't set. And at the top of the percentile range, that's the part that matters. That's the shape of the whole thing. You don't write a config file.

[00:03:58]You implement a trait and you get the request life cycle as code. If breakdowns like this are useful to you, there's a Patreon. Here's the part that barely got picked up. On May 4th, 2026, Cloudflare shipped a change log entry saying their cache now runs on a new proxy built on Pangora. Sit with that for a second because the significance is easy to miss. Pingora was the thing that connected Cloudflare to origin servers.

[00:04:20]Now, it's also the thing that decides what gets stored, what gets served stale, and what gets thrown away. The proxy absorbed the cache. Three behavior changes came with it. The first one is asynchronous stale while revalidate.

[00:04:33]Previously, the first request that arrived after an asset expired had to sit and wait for the origin to answer.

[00:04:38]That unlucky visitor paid for everyone else's freshness. Now that request gets the stale copy immediately, marked with a cache status of updating and the revalidation happens in the background.

[00:04:49]Worth being precise here. Cloudflare had started rolling this behavior out back in February. May is when it became native to the Pingora powered cache end to end. The second is unbuffered bypass on by default. Responses that bypass the cache are now streamed straight to the client instead of being buffered first.

[00:05:06]If the content was never cachable, holding it in memory before forwarding it was pure added latency. That's gone.

[00:05:13]The third is stricter RFC compliance. A very header set to asterisk now bypasses the cache entirely per RFC 9110. Set cookie headers get stripped on cache miss and expired responses for cachable assets. Floating point TTLs round down to the nearest integer. That third bucket looks like housekeeping. It isn't. Those are the edges where a cache quietly serves one user's response to another user. Getting stricter about the spec is a correctness decision wearing a change logs close. And the reported wins are architectural, not cosmetic. Lower per request overhead from better connection reuse and better origin offload from improved cash retention.

[00:05:50]Same idea as 2022. Fewer handshakes and less waste applied to a completely different layer. So what did Rust actually buy? Stripped of the evangelism. It bought predictability, not raw speed, predictability, the absence of a pause you didn't schedule.

[00:06:04]At 11 12 million requests a second, your median barely matters. Your 95th percentile is the product. A garbage collector is a tax you pay in exactly the percentile you can least afford. It bought a smaller machine. 70% less CPU on the same traffic is a line item on a data center budget and it bought the ability to keep going. Ngx didn't fail Cloudflare. It ran out of headroom. The per worker connection pool was a reasonable decision made for a different scale and Cloudflare grew past the assumption. Now the honest half because the balance matters more than the pitch.

[00:06:37]Rust is slower to write. The borrow checker turns runtime crashes into compile time arguments and those arguments cost you afternoons. The learning curve is real and it is steep and a team that's fluent in Go will ship a working service faster in Go, probably much faster, which makes the trade straightforward. You pay in developer time to buy machine time and tail latency guarantees. Cloudflare's traffic makes that trade obvious for most software. It's a genuinely bad deal. And pretending otherwise is how teams end up with a 2-year rewrite of a service that served 400 requests a minute. The signal is that the biggest operators on the internet will now pay in developer time for control over memory and latency. 10 years ago, they wouldn't. Cloudflare isn't alone here, and that's the actual story. Discord moved one hot path service off Go for the same reason.

[00:07:25]Garbage collection pauses. Microsoft has been rewriting parts of the Windows kernel in Rust, the boring, unhyped version of that story. Same pattern showing up in different buildings. Teams reaching down a layer and taking back control of memory because the abstraction they were standing on stopped being free. For years, the industry's default answer to slow software was more hardware. That answer stopped working. The rewrites are happening because the free lunch ended and the bill came due at the layer nobody wanted to touch. That's not one company's engineering decision. That's a pattern. There's a full breakdown of it on the channel, The Great Dloating, where we take apart why the whole stack is hitting this wall at the same time.

[00:08:03]Cloudflare just happens to be the clearest place to watch it