Cards/Distributed SystemsSystem Design · Day 96Nov 6, 2026

Every Distributed Bug Is A Time Bug

Write skew, a double charge, a vanishing comment and an expired lock set out as four failures with one shape — observe, a time gap in which the world changes, then act — and three fixes that rhyme

Ninety-five days of failures. Strip the technology away and most of them are the same shape.

Write skew: two transactions each read a valid world, both act, and the world they validated is gone by the time they commit. The double charge: the payment succeeded, the answer was lost coming back, and the retry acted on stale knowledge. The vanishing comment: the write landed on the primary and the read reached a replica still living 200ms in the past. The expired lock: a process paused, woke up, and carried on believing it held something it had lost twenty seconds earlier.

Four modules, four mechanisms, one enemy — the gap between observing and acting.

The fixes rhyme too. Make the check and the act atomic, so nothing can slip between them. Carry a token from the check — a version, an ETag, a timestamp — so the act can be refused if the world moved underneath it. Or let the resource refuse the past itself, through optimistic concurrency, compare-and-swap, a conditional update.

All three admit the same thing: by the time we act, the observation is already history.

We can't make the world hold still while we decide. We can only build things that notice we were looking at an older version of it.

We can't make the world hold still while we decide. We can only build things that notice we were looking at an older version of it.