A hot cache key expires. In the millisecond that follows, ten thousand requests miss.
All ten thousand ask the database. For the same row. At once.
The cache didn't fail. It worked exactly as designed — and that's what killed the database. This is a cache stampede, and it's most likely on your most popular data, at your busiest hour, which is precisely when you can least afford it.
Three defences:
→ Single flight. First request takes a lock, fetches, and fills. The other 9,999 wait for that one answer.
→ Jittered TTLs. Never expire together. Set 300 seconds ± random.
→ Serve stale, refresh behind. Return the expired value immediately, recompute in the background. Nobody waits, nobody piles up.
Note the third one: you deliberately serve slightly wrong data to stay alive. Same trade as the cart on Day 1, wearing different clothes.