39 long-form articles on production engineering — how real systems fail, why, and what actually fixed them.
22 articles matching “WAL”
Picking the Right DB22 min read
A senior-engineer walkthrough of Log-Structured Merge Trees: MemTable + WAL durability, SSTable layout with sparse index and bloom filters, the L0→Ln level hierarchy, leveled vs tiered vs FIFO compaction, the three…
Production Incidents45 min read
A payment commits, the API returns 200, the rack loses power, and after recovery the row does not exist. Before blaming the database, six questions have to be separated.
Architecture Decisions18 min read
Everyone can define read-your-writes consistency; far fewer can implement it on a master/replica cluster without melting the master.
Query Optimization18 min read
Every team ships with the same lie: throw everything on the database and it'll handle correctness. Works at 50 writes/sec.
Kafka & Pipelines25 min read
You add 24 more workers to fix Kafka lag — and they all sit idle. Meanwhile one huge message is stuck at the front of a partition, and nothing behind it moves.
Query Optimization13 min read
Every list endpoint ends in LIMIT 20 OFFSET ?. It has an index, page one returns in 8ms, and it passes review — then one Tuesday page 4,000 takes four seconds and a nightly export job stops finishing.
Query Optimization15 min read
We're trained to demand our data structures be correct — so here's one that's wrong on purpose, and runs inside Cassandra, RocksDB, Bigtable, most CDNs, and your browser's malware check.
Observability16 min read
The bell curve taught you the average is a fair summary and outliers are rare. In software both halves are false: latencies, key popularity, file sizes and follower counts are heavy-tailed power laws where the outliers…
Tradeoffs in Production15 min read
When a service buckles under load, the first instinct is to make the buffer bigger — and it's the most natural wrong answer in systems engineering.
Architecture Decisions16 min read
"Stamp each write with the current time and keep the latest" feels obvious — and it silently deletes data, because two machines can never agree on what time it is.
Architecture Decisions18 min read
Most engineers think a leader election "picks the best node." It doesn't — it gets every node to agree on the same node, and agreement under failure is consensus.
Observability10 min read
Every distributed system contains a quiet subsystem that decides which peers are still alive — and "send a ping, no reply, mark it dead" is wrong in a few interesting ways.
Architecture Decisions22 min read
A plain-language walkthrough of Pat Helland's 2007 classic "Life beyond Distributed Transactions" — the paper that named the patterns every large system already uses.
Architecture Decisions20 min read
Every process you run will be told to stop — the only question is whether it gets a request it can act on or a bullet it never sees coming.
WebSockets & Real-time20 min read
When you press Enter in Slack, the message does not travel over the WebSocket your client is holding open — it goes out as a separate HTTPS POST to chat.
Production Incidents18 min read
It's 3 AM. Inserts are failing with 'Duplicate entry 2147483647' — but you never inserted a duplicate. Welcome to one of MySQL's most quietly famous outages.
Tradeoffs in Production16 min read
Kernel TCP keep-alive probes the network path; app heartbeats prove the peer process is serving; HTTP Connection: keep-alive only reuses sockets.
Architecture Decisions20 min read
A senior-engineer walkthrough of the whole cache stack: CPU L1/L2/L3, TLB, OS page cache, DB buffer pool, in-process LRU (Caffeine), distributed cache (Redis/Memcached), HTTP cache + ETags, CDN edge, browser, DNS, reverse proxies, query plan cache.
Picking the Right DB15 min read
A senior-engineer walkthrough of how Redis really handles TTLs. Why TTL=0 doesn't free memory. The two strategies — lazy (on-access) and active (the random-sample-and-25%-threshold sampler).
Architecture Decisions20 min read
SVG-backed walkthrough: ring hotspots, successor overload when a node dies, and two ring generations during deploy.
Schema Design14 min read
A plain-English walkthrough: how B-tree inserts actually work, what a page split costs, why random UUID v4 keys bloat the index by 2× and blow the buffer pool.
Architecture Decisions20 min read
A beginner-friendly walkthrough: what “concurrent” means, what the C10K and C10M problems actually are, how the industry usually fixes them (event loops vs whole-system design), and what still goes wrong in production —…