39 long-form articles on production engineering — how real systems fail, why, and what actually fixed them.
39 articles
AI Agents18 min read
The Refund Agent will not refund until the carrier confirms the return. The Shipping Agent will not create a return label until the refund is approved.
Production Incidents45 min read
A payment commits, the API returns 200, the rack loses power, and after recovery the row does not exist. Before blaming the database, six questions have to be separated.
Production Incidents28 min read
A timeout is not a failure. Ten real ways idempotency keys break in payment systems — lost gateway responses, two tabs, racing retries, Redis evictions, expired keys, duplicate webhooks — with the fix for each, ten…
Kafka & Pipelines25 min read
You add 24 more workers to fix Kafka lag — and they all sit idle. Meanwhile one huge message is stuck at the front of a partition, and nothing behind it moves.
Scaling & Infra22 min read
Not a "what is serverless" post. The stuff you learn after the demo, from someone who's run it at scale.
Scaling & Infra20 min read
Not a "what is Lambda" post. A deep dive into how Lambda actually works: Firecracker microVMs and the Worker/placement fleet, how invocations get routed on the warm vs cold path, and what the INIT phase really does (and…
Query Optimization13 min read
“We sold 3 tickets but only had 1.” The code is inside a transaction, looks right, works a hundred times locally, and only breaks in production under load.
Query Optimization13 min read
Every list endpoint ends in LIMIT 20 OFFSET ?. It has an index, page one returns in 8ms, and it passes review — then one Tuesday page 4,000 takes four seconds and a nightly export job stops finishing.
Query Optimization15 min read
We're trained to demand our data structures be correct — so here's one that's wrong on purpose, and runs inside Cassandra, RocksDB, Bigtable, most CDNs, and your browser's malware check.
Observability16 min read
The bell curve taught you the average is a fair summary and outliers are rare. In software both halves are false: latencies, key popularity, file sizes and follower counts are heavy-tailed power laws where the outliers…
Tradeoffs in Production15 min read
When a service buckles under load, the first instinct is to make the buffer bigger — and it's the most natural wrong answer in systems engineering.
Architecture Decisions16 min read
"Stamp each write with the current time and keep the latest" feels obvious — and it silently deletes data, because two machines can never agree on what time it is.
Architecture Decisions18 min read
Most engineers think a leader election "picks the best node." It doesn't — it gets every node to agree on the same node, and agreement under failure is consensus.
Observability10 min read
Every distributed system contains a quiet subsystem that decides which peers are still alive — and "send a ping, no reply, mark it dead" is wrong in a few interesting ways.
Architecture Decisions22 min read
A plain-language walkthrough of Pat Helland's 2007 classic "Life beyond Distributed Transactions" — the paper that named the patterns every large system already uses.
Architecture Decisions18 min read
Everyone can define read-your-writes consistency; far fewer can implement it on a master/replica cluster without melting the master.
Architecture Decisions20 min read
Every process you run will be told to stop — the only question is whether it gets a request it can act on or a bullet it never sees coming.
WebSockets & Real-time20 min read
When you press Enter in Slack, the message does not travel over the WebSocket your client is holding open — it goes out as a separate HTTPS POST to chat.
Production Incidents18 min read
It's 3 AM. Inserts are failing with 'Duplicate entry 2147483647' — but you never inserted a duplicate. Welcome to one of MySQL's most quietly famous outages.
Production Incidents25 min read
Why “our queue guarantees exactly-once” is almost always a lie once your flow crosses a component boundary: Kafka EOS scope, ack-after-process, unique constraints, SQS FIFO, DB transactions with SMTP, retries without…
Tradeoffs in Production16 min read
Kernel TCP keep-alive probes the network path; app heartbeats prove the peer process is serving; HTTP Connection: keep-alive only reuses sockets.
Architecture Decisions20 min read
A senior-engineer walkthrough of the whole cache stack: CPU L1/L2/L3, TLB, OS page cache, DB buffer pool, in-process LRU (Caffeine), distributed cache (Redis/Memcached), HTTP cache + ETags, CDN edge, browser, DNS, reverse proxies, query plan cache.
Picking the Right DB15 min read
A senior-engineer walkthrough of how Redis really handles TTLs. Why TTL=0 doesn't free memory. The two strategies — lazy (on-access) and active (the random-sample-and-25%-threshold sampler).
Picking the Right DB22 min read
A senior-engineer walkthrough of Log-Structured Merge Trees: MemTable + WAL durability, SSTable layout with sparse index and bloom filters, the L0→Ln level hierarchy, leveled vs tiered vs FIFO compaction, the three…
Query Optimization18 min read
Every team ships with the same lie: throw everything on the database and it'll handle correctness. Works at 50 writes/sec.
Architecture Decisions20 min read
SVG-backed walkthrough: ring hotspots, successor overload when a node dies, and two ring generations during deploy.
Schema Design14 min read
A plain-English walkthrough: how B-tree inserts actually work, what a page split costs, why random UUID v4 keys bloat the index by 2× and blow the buffer pool.
Load Balancing15 min read
A simple, ground-up guide: L4 forwards raw TCP/UDP by IP and port, L7 opens the HTTP request and routes by URL, host, headers or cookies.
Architecture Decisions20 min read
A beginner-friendly walkthrough: what “concurrent” means, what the C10K and C10M problems actually are, how the industry usually fixes them (event loops vs whole-system design), and what still goes wrong in production —…
Scaling & Infra20 min read
Every type of scaling explained simply — horizontal, vertical, reactive, predictive, scheduled. Why HPA and ASG break on Dream11 during an IPL final, flash sales, and push notifications.
Production Incidents16 min read
When the Garbage Collector pauses, every thread freezes — zero requests, zero responses. This post explains how GC actually works: the heap, mark-and-sweep, minor vs major collections, Stop-the-World pauses, and what…
Performance14 min read
Why your connection pool config is probably wrong — and how to fix it with Little's Law, Kingman's Formula, and the process-to-core ratio.
Schema Design12 min read
Why DELETE is the most expensive DML operation in a relational database — and why a simple UPDATE SET deleted_at = NOW() is 4-8x faster under concurrent load.
Query Optimization18 min read
A deep-dive into non-SARGable SQL with B-tree diagrams, real EXPLAIN output, and the exact rewrites — told through a gaming dashboard that went from 4,200ms to 2ms without touching the schema.
Production Incidents15 min read
A user had access to 1 lakh devices across MongoDB, ClickHouse, and Elasticsearch. We tried seven approaches. All broke. The full journey — dead ends, edge cases, and what production systems actually do.
Query Optimization13 min read
How similarity search works under the hood — and why naive brute-force collapses at scale.
Query Optimization5 min read
What happens when your index outgrows your data — and how we fixed it without downtime in production.
Architecture Decisions8 min read
Fan-out on write vs fan-out on read — and why Twitter uses a hybrid at 500M users. A beginner-friendly breakdown.
WebSockets & Real-time4 min read
A production case study — how a small config issue brought down an entire IoT device fleet and what fixed it.