Cards/KafkaSystem Design · Day 23Aug 23, 2026

The Consumer That Fell Behind Forever

The Consumer That Fell Behind Forever — system design card, day 23, kafka

Producers write 10,000 events a second. Your consumer handles 8,000.

Nothing errors. Nothing alerts. The gap just grows — 2,000 events a second, quietly, all day.

This is consumer lag, and it's the most under-watched number in event-driven systems. Kafka won't complain: the log is doing exactly its job, retaining everything. Your consumer is simply reading a version of the world that's getting older by the minute.

By evening you're processing lunchtime.

Then retention expires, and the events you never read are deleted. Now it's data loss.

The instinct is "add consumers." That only works up to your partition count — one partition feeds one consumer in a group. Six partitions, six consumers, done. The seventh sits idle.

So the real levers are: more partitions, faster processing, or fewer events.

The dangerous failures don't throw errors. They just fall a little further behind, every second.