How Uber serves 40 million reads per second

[00:00:00]One of the most fascinating problems in software development is that once a product reaches a certain scale, the biggest technical challenge is often no longer hackers or cloud outages, but handling your own traffic. And there are very few SAS companies which face the traffic pressure of Uber. You might be pretty familiar with Uber now that apparently the software developer job is not safe anymore and we are all looking for a backup. So the Uber team has recently published a detailed engineering post about how they protect their databases from overload. So in this episode of the blueprint, we'll look at how a tech giant handles distributed systems in production. For a bit of context, Uber runs thousands of microservices serving more than 170 million monthly active users. Underneath all of that sit two in-house distributed databases called Docto and Schmales, both built on top of good old reliable MySQL. Together they spend thousands of clusters, store tens of pabytes of operational data, and serve tens of millions of requests per second. At this scale, if one partition gets slow, clients time out, timeouts trigger retries, retries multiply the load, and a brief spike in one corner of the system turns into a platformwide incident. In the industry, this is called the thundering herd. So, in the next minutes, we look at how Uber went from static rate limiting to what is essentially a self-regulating immune system for their databases, and why the final solution relies on a piece of technology that predates the computer by about 200 years. But first, a quick history lesson because Uber's database story comes with a bit of drama. Back in 2014, Uber needed a solution to store the trip related data. Think of pickup, drop off, fair calculation, billing, adjustments, driver and rider feedback and all the tiny pieces of state that show up during or after a ride. The team looked at existing databases like Cassandra and MongoDB, but their requirements went beyond scaling capabilities. They needed a tool that could accept rights during failure, notify downstream systems when data changes, and support secondary indexes.

[00:01:57]Of course, they couldn't find the perfect solution. So, they did what all developers do best. They built schemalis, an in-house appendon data store designed to move its core trip data away from a single postgress instance and onto something that could scale horizontally. Then in 2016, Uber published the now infamous post explaining why they moved from Postgress to MySQL. The core issue was right amplification. This is a bit technical, but it is enough for you to know that in Postgress, an update usually creates a new row version instead of modifying the existing row in place. Because secondary indexes point to physical tuple locations, even a small update can require new index entries, more write ahead log rights, which is the database's durability journal by the way, more disio, and more cleanup work later through vacuum. This became especially expensive for Uber because their trip data was heavily indexed and frequently updated. Replication made the problem worse. At the time, Postgress replication was mainly physical while replication. So, replicas received a low-level storage changes, not compact logical operations. That meant local right amplification also turned into replication bandwidth and replica lag problems, especially across data centers. As a result, Uber preferred MySQL with InnoDB because secondary indexes reference the primary key rather than the physical row location that reduced the amount of index maintenance needed for some update patterns and made the system fit their workload better.

[00:03:21]Again, I know that this is very technical, but I find it really fascinating. At scale, such small implementation details matter a lot.

[00:03:29]Then in 2020, their initial schemalis database solution evolved into doctore.

[00:03:34]This is a generalpurpose transactional database built on top of the MySQL storage platform which runs essentially every business vertical at Uber today ranging from rightes to payments to food delivery. Architecturally both databases share the same skeleton. There is a stateless query engine on top which handles routing, sharding, parsing and authorization. Below you'll find the stateful storage engine where data is sharded across partitions. Each partition consisting of one leader and two followers coordinated through rough consensus all backed by MySQL. So despite the exotic names, the foundation is actually boring old technology and principles used in clever ways. But now let's address the overload problem. The first thing Uber tried was a quotota based rate limiting approach in the query engine layer. Assign every request a capacity cost based on bytes processed. Give every user a quotota.

[00:04:24]Return a 429 when they exceed it. This is simple, clean, efficient, and apparently completely wrong. Since the rooting nodes are stateless, quota usage had to live in a central cache, which means they added a database call to every request in order to protect the database. This is a new network hop and a new single point of failure purchased specifically to improve reliability.

[00:04:46]Second, the cost model was broken in a way that is almost funny because of how my SQL handles scanning and filtering. A query reading a single table row was built exactly the same as a query doing a full table scan. So the metering system could not tell the difference between a feather and a piano as long as both were delivered in one box. And third, the quotas were static, which in a multi-tenant system means an endless procession of teams filing tickets to politely ask for more quota, which they always received, which made the mandatory quota fully optional. So the whole thing failed, but it produced the insider their next step. The obvious conclusion was that overload management has to live as close to the storage nodes as possible because that is the only place with full context about what the system is actually feeling. Also, they learned that counting queries per second is to course because not all queries are equal. What actually reflects load is concurrency, the number of operations currently in flight, which by little's law equals throughput times latency. So Uber's first real load sheddder was built around control delay, an algorithm borrowed from networking where it was invented to fight buffer bloat. Instead of watching how long the queue is, Codel watches how long requests wait in it. They rent separate cues for reads, writes, and slow background operations and added a clever twist. Under normal load, the queue behaves as FIFO, but under pressure, it flips to LIFO. That's because during an overload, the oldest requests in the queue have usually already been abandoned by their clients who have long since timed out and retried. So processing them is most of the time useless. The newest requests, however, are the ones that still have a living waiting client on the other end. So you serve those instead. They added a rule-based admission system called scorecard to cap per tenant concurrency.

[00:06:30]So a single noisy neighbor cannot monopolize shared infrastructure plus a set of node local regulators watching for the sneakier failure modes like right volume saturating IO traffic hammering one hot partition key low memory and even a regulator that counts go routines and starts throttling when there are too many of them. This setup kept the databases alive but it had one fatal flaw. It was completely blind to priority. When overload hit, Kodell shed traffic indiscriminately, meaning a real-time ride request and some internal analytics aggregation job had exactly the same odd of survival. Its timeouts were also static around 5 milliseconds.

[00:07:06]So when shedding kicked in, everything got rejected at once. Everything retrieded at once and the system oscillated between total rejection and total acceptance. Which brings us to the current design and to my favorite detail in this entire story. Uber replaced Kodell with Cinnamon, a priority aware load shedter originally built by their delivery team. Every request carries a priority tier from tier zero for critical infrastructure down to tier five for workloads nobody will cry about. During overload, Cinnamon sheds from the bottom up. So, your ride request keeps working while someone's batch pipeline quietly eats a server error response. But the interesting part is how Cinnamon decides how much to shed. Instead of fixed thresholds, it uses a P controller. That's proportional, integral, derivative, a control loop mechanism whose mathematical formulation dates to 1922 when an engineer named Nicholas Minorski worked it out by watching helmsman steer ships for the US Navy. The same feedback logic that keeps your oven at 180° and kept battleships on course a century ago now decides which database requests live and which ones die for your Uber ride. A P controller doesn't just react to the current error. It incorporates history and trend. Uber describes the difference as a hammer versus a dimmer switch.

[00:08:21]Token bucket limiter slam the door, cause retry storms, and pileup memory.

[00:08:25]The P controller smoothly converges on a stable rejection rate and hold it there.

[00:08:30]The final evolution turns Cinnamon into a general purpose overload engine with pluggable signals, which Uber calls bring your own signal. Yes, I know they like creating tools and naming things.

[00:08:41]This matters because in a distributed system, overload isn't always local. A perfectly healthy leader node may still need to shed traffic because its rough followers are falling behind on replication. Now that commit-like signal plugs into the same decision loop as memory pressure and concurrency. So all shedding decisions come from a single brain. The results are 80% more throughput under overload, coroutine count during incidents down 93% and heap usage down from 5 or 6 GB spikes to about 1 GB. All from replacing static thresholds with a feedback loop from 1922. And before wrapping up, here is your awesome trivia of the day.

[00:09:19]>> On January 15th, 1990, AT&T's long-distance network collapsed for 9 hours. A single switch in Manhattan performed a routine self-reset and broadcast an outofservice message to its neighbors. Those neighbors contained a bug that made them crash when processing messages arriving too quickly. Each crashing switch broadcast its own out of service message, crashing more neighbors, which then recovered and got crashed again by the recovering switches around them. Around half of all long-distance calls in America failed that day, costing AT&T an estimated $60 million. The entire network was healthy and the hardware was fine, but the system destroyed itself with its own recovery traffic, which is exactly the failure mode Uber spent a decade engineering against. Please let me know in the comments if you enjoy these architecture deep dives. Don't forget to like the video, subscribe to the channel, and until next time, thank you for watching.