How coordinated timeouts helps in preventing cascading failures

本站添加: 2026-05-27

[00:00:00]Another important protection against cascading failures is coordinated timeouts. Say timeouts are good because they stop a service from waiting forever, right? But timeouts must be designed carefully across the whole call chain. As you can see on the screen, service A has 3 second timeout. Service B has a 2C timeout. Service C has a 5-second timeout. And let's say service D is slow. The problem happens when the timeout of an inner service is longer than the timeout of its caller. So for example, service B may wait only 2 seconds for service C. But service C may wait 5 seconds for service D. That is a problem because after 2 seconds, service B has already given up. It has set timeout. From service B's point of view, the request has failed. But service is still waiting for service D for another 3 seconds. So service C is now wasting CPU, memory, thread, etc. on a request that nobody upstream is waiting for anymore because service B has given up already. This is how resources get wasted silently inside the system. And when thousands of requests behave like this, imagine the wasted weights how they can pile up and how they can create some sort of an overload. So the rule is very simple. As we go do deeper into the call chain, timeout should usually get smaller and not larger.