Tiny script massive outage

本站添加: 2026-05-16

[00:00:00]This tiny maintenance script brought the entire API to its knees.

[00:00:03]>> [music] >> A sleepy Sunday cron ran a harmless-looking SQL update across a massive [music] table. Everything was fine until latency spiked and requests started queuing. Load balancers were happy, app servers looked healthy, but the DB was choking. People assumed a cache issue or runaway worker. So, we restarted everything. Restarts [music] did nothing because the database was blocked by a single long transaction.

[00:00:32][music] The culprit was one maintenance query that forgot a where clause and started rewriting millions of rows.

[00:00:39]>> [music] >> Because it ran in a single transaction, every write waited for it to finish and eventually timed out.

[00:00:44]We killed the job, but the rollback was massive [music] and took hours, keeping the system degraded. The simple fix, run updates in small [music] batches and avoid long transactions on hot tables. Impact, [music] minutes of downtime turned into hours, thousands of failed requests, [music] and a very angry dashboard. Comment what you think we missed and watch again because there's an even worse version of this mistake.