How Load Balancing Protects Your Cluster from Individual Server Crashes

Added: 2026-05-16

[00:00:00]But what happens when the data set gets so big that one machine physically can't hold it? The obvious answer everyone reaches for is modulo. You hash the key, take the result modulo the number of servers, and that number tells you where the key lives. With three servers, it works fine. Keys spread out and everything runs smoothly. Then you need to add a fourth server. The formula changes from modulo three to modulo four, and almost every single key remaps to a different server. Every one of those remapped keys is now a cache miss, going straight to the database and at scale. That's not just a slow down.

[00:00:29]That's everything falling over at once.

[00:00:31]So you need something smarter. That smarter thing is the hash ring. Instead of a flat modulo formula, you take your entire hash space and wrap it into a circle. A ring that goes from zero all the way around back to zero. And the one rule is that you always move clockwise.

[00:00:44]You hash each server, and wherever it lands on the ring, that's its spot.

[00:00:48]Server A here, server B here, server C here, just points sitting on a circle.

[00:00:52]When a key comes in, you hash it, find where it lands, walk clockwise until you hit a server, and that server owns the key. That's the whole look up. No formula to break, no modulo to recalculate. When server D joins, it drops in between two existing servers, and only the keys in that new stretch of the ring move over. Just one key out of four in our example. Whereas with modulo, every single key would have moved. When a server crashes, its keys flow clockwise to the next server on the ring, and only that server's keys move.

[00:01:17]Everyone else stays completely untouched. One catch is that with just a few servers, the gaps between them are rarely even. So one server can end up responsible for way more keys than the others. The fix is virtual nodes, where instead of placing each server once on the ring, you place it multiple times at different positions spread around. Load balances out naturally, regardless of where servers physically ended up. This is what Amazon DynamoDB runs on, Cassandra too, CDNs, load balancers, all of it. Thousands of real nodes, millions of virtual positions, billions of keys all finding their home. You add a server, and almost nothing moves. You lose a server, and almost nothing moves.

[00:01:51]The ring just absorbs it.