39 long-form articles on production engineering — how real systems fail, why, and what actually fixed them.
Production incidents, real-time systems, and Kafka pipelines — what broke and how it was fixed.
6 articles in Production Incidents
Production Incidents45 min read
A payment commits, the API returns 200, the rack loses power, and after recovery the row does not exist. Before blaming the database, six questions have to be separated.
Production Incidents28 min read
A timeout is not a failure. Ten real ways idempotency keys break in payment systems — lost gateway responses, two tabs, racing retries, Redis evictions, expired keys, duplicate webhooks — with the fix for each, ten…
Production Incidents18 min read
It's 3 AM. Inserts are failing with 'Duplicate entry 2147483647' — but you never inserted a duplicate. Welcome to one of MySQL's most quietly famous outages.
Production Incidents25 min read
Why “our queue guarantees exactly-once” is almost always a lie once your flow crosses a component boundary: Kafka EOS scope, ack-after-process, unique constraints, SQS FIFO, DB transactions with SMTP, retries without…
Production Incidents16 min read
When the Garbage Collector pauses, every thread freezes — zero requests, zero responses. This post explains how GC actually works: the heap, mark-and-sweep, minor vs major collections, Stop-the-World pauses, and what…
Production Incidents15 min read
A user had access to 1 lakh devices across MongoDB, ClickHouse, and Elasticsearch. We tried seven approaches. All broke. The full journey — dead ends, edge cases, and what production systems actually do.