Cards/DatabasesSystem Design · Day 46Sep 17, 2026

Splitting a Table While People Are Using It

A 2TB table moving onto four shards with no downtime through dual write, backfill, verify, gradual read shift and finish, set against the naive stop-and-copy plan

The table is 2TB and growing. It needs to live on four machines instead of one. Business hours don't stop.

The naive plan is to stop writes, copy everything, and point the app at the new shards. Hours of downtime, high risk, and no easy way back if something is wrong at the end of it.

The plan that works has five stages. Dual write — every write goes to old and new, reads still come from old. Backfill — copy history in batches while live traffic continues. Verify — read from both and compare, and don't skip this, because it is where you learn what the backfill missed. Shift reads — 1% to 10% to 100%, one dial, reversible at every step. Finish — stop dual-writing, and keep the old copy for a while anyway.

It takes weeks rather than an evening. Nothing is ever unrecoverable.

The expensive part of a big migration isn't moving the data. It's never being in a state we can't retreat from.

The expensive part of a big migration isn't moving the data. It's never being in a state we can't retreat from.