The load test says p99 is 40ms. Production says 900ms. The benchmark was lying — politely, and by design.
A naive load generator sends the next request only after the previous one completes. When the server stalls for two seconds, the generator stalls with it. At 100 requests a second that is two hundred requests which should have been sent and never were — and every one of them would have been a terrible measurement. The worst numbers are missing, so the percentiles look wonderful.
This is coordinated omission: the measurement coordinates with the thing it is measuring.
The fix is to send on a schedule and measure against that schedule. A request is late from the moment it was due, whether or not we managed to send it. Tools like Vegeta, k6 and Locust can hold a constant arrival rate, and the same two-second stall then surfaces as a p99 of 900ms — which is what the user was experiencing all along.
A measurement that pauses when the system struggles will always report a system that doesn't struggle.