39 long-form articles on production engineering — how real systems fail, why, and what actually fixed them.
13 articles matching “Distributed Systems”
Architecture Decisions16 min read
"Stamp each write with the current time and keep the latest" feels obvious — and it silently deletes data, because two machines can never agree on what time it is.
AI Agents18 min read
The Refund Agent will not refund until the carrier confirms the return. The Shipping Agent will not create a return label until the refund is approved.
Architecture Decisions22 min read
A plain-language walkthrough of Pat Helland's 2007 classic "Life beyond Distributed Transactions" — the paper that named the patterns every large system already uses.
Production Incidents45 min read
A payment commits, the API returns 200, the rack loses power, and after recovery the row does not exist. Before blaming the database, six questions have to be separated.
Production Incidents28 min read
A timeout is not a failure. Ten real ways idempotency keys break in payment systems — lost gateway responses, two tabs, racing retries, Redis evictions, expired keys, duplicate webhooks — with the fix for each, ten…
Kafka & Pipelines25 min read
You add 24 more workers to fix Kafka lag — and they all sit idle. Meanwhile one huge message is stuck at the front of a partition, and nothing behind it moves.
Architecture Decisions18 min read
Most engineers think a leader election "picks the best node." It doesn't — it gets every node to agree on the same node, and agreement under failure is consensus.
Observability10 min read
Every distributed system contains a quiet subsystem that decides which peers are still alive — and "send a ping, no reply, mark it dead" is wrong in a few interesting ways.
WebSockets & Real-time20 min read
When you press Enter in Slack, the message does not travel over the WebSocket your client is holding open — it goes out as a separate HTTPS POST to chat.
Production Incidents25 min read
Why “our queue guarantees exactly-once” is almost always a lie once your flow crosses a component boundary: Kafka EOS scope, ack-after-process, unique constraints, SQS FIFO, DB transactions with SMTP, retries without…
Architecture Decisions20 min read
SVG-backed walkthrough: ring hotspots, successor overload when a node dies, and two ring generations during deploy.
Schema Design14 min read
A plain-English walkthrough: how B-tree inserts actually work, what a page split costs, why random UUID v4 keys bloat the index by 2× and blow the buffer pool.
Production Incidents15 min read
A user had access to 1 lakh devices across MongoDB, ClickHouse, and Elasticsearch. We tried seven approaches. All broke. The full journey — dead ends, edge cases, and what production systems actually do.