Engineering notes from the real world
Deep dives into the systems we build, the failures we encounter, and the trade-offs that make production engineering difficult.
The Refund Agent will not refund until the carrier confirms the return. The Shipping Agent will not create a return label until the refund is approved. Each rule is correct.
AI Agents18 min read
Read the articleProduction Incidents45 min read
A payment commits, the API returns 200, the rack loses power, and after recovery the row does not exist. Before blaming the database, six questions have to be separated.
Production Incidents28 min read
A timeout is not a failure. Ten real ways idempotency keys break in payment systems — lost gateway responses, two tabs, racing retries, Redis evictions, expired keys, duplicate webhooks — with the fix for each, ten…
Kafka & Pipelines25 min read
You add 24 more workers to fix Kafka lag — and they all sit idle. Meanwhile one huge message is stuck at the front of a partition, and nothing behind it moves.
Scaling & Infra22 min read
Not a "what is serverless" post. The stuff you learn after the demo, from someone who's run it at scale.
Scaling & Infra20 min read
Not a "what is Lambda" post. A deep dive into how Lambda actually works: Firecracker microVMs and the Worker/placement fleet, how invocations get routed on the warm vs cold path, and what the INIT phase really does (and…
Query Optimization13 min read
“We sold 3 tickets but only had 1.” The code is inside a transaction, looks right, works a hundred times locally, and only breaks in production under load.
Schema design, query optimization, storage engines, and picking the right database.
Production incidents, real-time systems, and Kafka pipelines — what broke and how it was fixed.
Load balancing, messaging, Lambda, and scaling infrastructure on AWS.
Architecture decisions, distributed-systems trade-offs, and observability.
Exploring LLMs, RAG, AI agents, memory, evaluations, and real-world AI engineering.
I'm Amit Verma, a software engineer working on distributed systems — Kafka pipelines, WebSocket infrastructure, and the databases underneath them.
This publication exists because most content shows the clean answer. Here you get the real problem: the production failure, the approaches that didn't work, the technical decision, and the lesson that stuck.