Why "Just Add a Queue" Never Fixes Overload | Backpressure & Load Shedding Explained

[00:00:00]Here is a fix that betrays almost everyone the first time. Your service is falling over under load, so you do the obvious thing, put a queue in front of it. Requests come in, they wait their turn, the workers pull from the queue.

[00:00:13]It feels like you added a shock absorber. Watch what actually happens on the screen when the overload is not a brief spike, but sustained. Requests arrive at a thousand a second, your workers drain 200 a second, the queue does not smooth that out. It just grows because arrivals outrun the drain rate forever. And now every new request lands at the back of an enormous line of work that is already stale. By the time a worker finally picks up a request, the client that sent it gave up long ago.

[00:00:42]You burn CPU producing an answer nobody is waiting for. Fred Hebert put it exactly, a queue does not fix overload, it hides it and then hands you the same failure later, slower and more expensive. The queue bought you nothing but latency. So what is the thing a queue is missing?

[00:00:59]There is a law that makes this precise, and it is one every engineer should keep in their pocket, Little's Law.

[00:01:06]The number of items in a system equals the average arrival rate times the average time each item spends inside.

[00:01:14]Rearrange it and the trap is obvious. If the arrival rate is higher than the rate you can service, the only free variable left is time. So the time each request spends waiting climbs without limit, and the queue length climbs with it. An unbounded queue does not absorb sustained overload. It silently converts overload into ever-growing latency, right up until you run out of memory or every client times out. This is why just make the queue bigger is not an answer.

[00:01:42]A bigger queue just means a longer wait before the same collapse. Queues are genuinely useful, but only for what they are, shock absorbers for short bursts that you can drain between them.

[00:01:52]For a burst that lasts 2 seconds when you have slack, a queue is perfect. For a load that sustains above your capacity. There is no queue size large enough because the problem is not buffering, it is arithmetic. So, if buffering cannot save you, what can?

[00:02:08]The thing the naive queue was missing is a signal that goes the wrong way.

[00:02:12]Normally, data flows forward, producer to buffer to consumer. Backpressure is the flow of information backward from the overloaded stage upstream to whoever is feeding it, carrying one message, slow down or stop. And the mechanism that makes it real is almost embarrassingly simple. The buffer must be bounded. Give it a hard limit.

[00:02:34]Now, when the consumer cannot keep up, the buffer fills, and the fullness itself is the signal.

[00:02:40]A producer trying to push into a full bounded buffer gets told no. It blocks, it slows, or it is refused outright.

[00:02:48]Compare the two worlds.

[00:02:50]With the unbounded queue, overload was invisible. Everything looked like it was accepted, and the damage showed up minutes later as latency and timeouts.

[00:03:00]With a bounded buffer, overload is explicit and immediate. The system says, "I am full right now." to the exact component that can do something about it. You have converted a silent, delayed collapse into a loud, honest, actionable no. But, what should the system actually do the moment the buffer says no?

[00:03:19]When the bounded buffer is full, pretending is no longer an option, and you have exactly four honest responses.

[00:03:25]One, block the producer. Make it wait until there is room. Inside a single process, between threads, this is ideal.

[00:03:33]The pressure propagates naturally up the call chain until it reaches something that can genuinely slow down, like a thread reading from a socket. Across a network, blocking is dangerous. If you just make the caller wait, you have not removed the queue, you have moved it into the caller, and now their buffers fill, too. Two and three, drop work.

[00:03:54]Either shed the newest arrivals or sometimes smarter, drop the oldest items that have been waiting longest and are most likely already stale. Four, reject at the door. Refuse the request immediately with a clear signal before it ever consumes a resource.

[00:04:09]Options two, three, and four are all the same discipline under one name, load shedding.

[00:04:15]You deliberately, surgically fail some requests right now, cheaply, so that the requests you do accept actually succeed.

[00:04:22]It sounds like giving up. It is the opposite. It is the only way to stay alive under load you cannot serve. So, how do you shed load without the shedding itself becoming the outage?

[00:04:32]Load shedding done wrong becomes the incident.

[00:04:36]If you only discover you are overloaded after a request has opened a database connection, grabbed a thread, and called three downstream services, then rejecting it wasted all of that and the rejection itself is expensive. So, the rule is shed at the edge as early and as cheaply as possible.

[00:04:54]The very front door. Admission control checks a fast signal of health and if the system is saturated, turns the request away before it touches anything expensive. And how you say no matters enormously.

[00:05:07]Do not let the request hang for 30 seconds and then fail. That holds a connection, burns a thread, and teaches the client nothing. Return immediately with an explicit rejection. In HTTP terms, a 503 service unavailable.

[00:05:21]Ideally with a retry after header telling the client how long to wait before trying again. This is the exact lesson from the circuit breaker episode.

[00:05:30]Now, on the receiving side, a clean instant no is infinitely more valuable than a slow, resource-eating maybe. Fail fast, fail cheap, fail informatively.

[00:05:41]But, if you have to turn away, say 10% of traffic, which 10% should it be?

[00:05:47]If you must shed, shedding at random throws away a checkout as readily as a background prefetch, a waste. The mature move is to make shedding a choice ranked by value. Google's SRE book formalizes this with request criticality. Every request carries a label. Is it critical user-facing work or is it sheddable?

[00:06:06]Something the system would prefer to serve but can survive dropping.

[00:06:11]When you are overloaded, you shed from the bottom of that ranking up. Drop the batch refreshes and the speculative prefetchers first. Protect the logins and the payments till the last. The same idea guards you against a nasty feedback loop. Remember from the circuit breaker episode, a rejected request that immediately retries is not relief. It is brand new load arriving at the worst possible moment, the retry storm. So, retries must be disciplined. A retry budget that caps retries as a fraction of traffic and back off with jitter. So, a thousand clients do not all retry in lockstep. Shed the cheap stuff, protect the valuable stuff, and make sure your own clients are not amplifying the overload. Now, one-hop shedding is good but the queue trap can hide anywhere in a chain. How do you make the whole pipeline honest?

[00:06:59]A single service shedding correctly is not enough because a pipeline is only as honest as its most forgiving stage. If stage A pushes back properly but feeds stage B through an unbounded queue, you have simply relocated the lie. The overload hides in B's queue instead. So, the discipline has to be end-to-end.

[00:07:19]Every buffer in the whole chain is bounded and the pressure propagates from the slowest stage all the way back to the true source.

[00:07:26]This is exactly what the reactive streams standard turned into a protocol, the backbone of libraries like project reactor and Akka Streams. Instead of the producer pushing whenever it likes, the consumer builds its signals I am ready for n more items and the producer is contractually forbidden from sending more than we asked for.

[00:07:47]Demand flows upstream. Data flows downstream only as fast as demand allows. The buffer can never overflow because nobody is allowed to overfill it. The crucial part is that the signal must reach the actual origin, the network socket, the message queue polar, the upstream client because if it only reaches the second to last stage, that stage becomes the new unbounded reservoir. Push the pressure all the way home. So, where does all of this fit and where does it bite?

[00:08:17]So, where do these tools belong? Put a bounded buffer between every producer and consumer that can run at different speeds, thread to thread, service to service, stage to stage and choose the bound deliberately. Too small and you reject work during normal healthy bursts, hurting yourself for no reason.

[00:08:33]Too large and you drift back toward the unbounded trap, absorbing overload into latency before you finally fail. The right size is small enough to detect trouble fast, large enough to ride out the burst you actually see in production, a number you tune from real traffic, not a guess and there is an honest price to all of this. Back pressure and load shedding mean that under genuine overload, you will reject valid requests from real users who did nothing wrong. That can feel like failure, but it is the opposite. It is the system admitting out loud that its capacity is finite and choosing to keep its promises to the requests it accepts rather than breaking them for everyone.

[00:09:14]Keep one image in mind as the test. A queue is a shock absorber, not a reservoir. A shock absorber is empty most of the time and compresses under a bump. If your queue is permanently full, it is not buffering anything. It is a capacity problem wearing a disguise.

[00:09:30]What is the checklist you carry home?

[00:09:33]The portable checklist, five rules. One, bound every buffer, no unbounded queues anywhere, ever. An unbounded queue is not a safety margin. It is a hidden countdown to a slower, worse failure.

[00:09:46]Two, turn fullness into an upstream signal. A full bounded buffer should push back on its producer, not silently swell. So, overload becomes visible at the moment and place it happens. Three, shed early, cheap, and at the edge.

[00:10:00]Decide to reject before the request consumes a thread, a connection, or a downstream call. And say no with a fast, explicit rejection the client can act on.

[00:10:09]Four, shed the least valuable work first. Rank requests by criticality and protect the work that matters and give your own clients retry budgets and back off so their retries do not become the next wave of overload.

[00:10:22]Five, remember that a permanently full queue is a capacity problem, not a buffer. Buffering is for bursts, and if the queue never drains, no amount of tuning will save you from needing more capacity or less load. Five rules, one theme, be honest about your limits out loud in real time. Let us land the whole story.

[00:10:43]Landing it. The trap that opens this story is the most natural fix in the world. A service is overloaded, so you add a queue, and it fails because a queue without a limit does not absorb overload, it stores it. Converting a capacity problem into unbounded latency and then a wall of timeouts with your workers burning effort on requests nobody is waiting for anymore.

[00:11:05]The thing the queue lacked was a signal that flows the other way, back pressure, the message a bounded buffer sends upstream the moment it fills saying, "Slow down."

[00:11:15]Give every buffer a hard limit and overload stops being invisible and delayed. It becomes explicit and immediate, aimed at the one component that can react. When you genuinely cannot keep up, you shed load deliberately, cheaply, at the edge with a fast 503 instead of a slow hang, and you shed the least valuable work first, protecting the checkouts and the logins while dropping the pre-fetches.

[00:11:40]All while keeping your own retries on a budget, so they do not become the next surge. And you make it end-to-end, every stage bounded, and the pressure reaching all the way back to the true source.

[00:11:51]Because one unbounded buffer anywhere re-hides the whole problem. The discipline to keep a queue is a shock absorber, not a reservoir. Be honest about your capacity in real time out loud. Next in the series, we stay with distributed data. Thanks for watching.