In 2007 Pat Helland — a man who spent a large chunk of his career building systems that promised global serializability — wrote a paper. In it he confessed that he no longer believed in them. He called it “an Apostate’s Opinion.” Almost twenty years later it reads less like a confession and more like a blueprint for how every large system you use is actually built.
This post explains that paper in plain language. It connects each idea to the stack you ship today — sharded databases, Kafka, microservices, idempotency keys, sagas. It gives the vocabulary a name where the paper coined one. The two words worth remembering are entity and activity. Everything else falls out of them.
The Maginot Line
Helland opens with a metaphor. The Maginot Line was a vast, expensive fortification France built along its German border between the two World Wars. It worked exactly as designed: the German army never crossed it directly. In 1940 they simply drove around it, through Belgium.
His claim is that distributed-transaction protocols — two-phase commit (2PC), and the quorum/consensus machinery around it — are the Maginot Line of our industry. They are technically magnificent. They give the application programmer a beautiful façade: global serializability. It is the illusion that the entire system is one big database. In that database you can atomically update any set of rows on any set of machines.
And in practice, builders of very large systems route around them. Not out of ignorance — out of natural selection. When teams depend on distributed transactions at scale, the projects tend to founder on two rocks. The first is performance, because every cross-node commit pays a coordination tax. The second is fragility. 2PC blocks when a participant or the coordinator is unavailable. And availability is the thing you were scaling for in the first place.
The thought experiment: almost-infinite scaling
The frame for the whole paper is what Helland calls almost-infinite scaling. Imagine the number of customers, orders, shipments, accounts, patients, devices — every business noun your app manages — growing without bound. Tens of thousands, hundreds of thousands of machines. Too many to pretend they are one big machine.
The key observation: the individual things usually do not grow. One order is still one order. You just get more of them. So the system spreads what used to fit on a few machines across a great many machines. We want it to scale roughly linearly with load, in both data and computation.
This forces a precise question that small systems never have to ask:
When can I know that two pieces of data live on the same machine? And what do I do when I cannot guarantee they do?
That question is the hinge of everything that follows.
Two layers: scale-agnostic and scale-aware
Helland’s first assumption is that a scalable application has (at least) two layers that perceive scaling differently.
- The lower layer is scale-aware. It knows that the cluster gains machines. It owns the mapping from logical data to physical locations. As load grows, it repartitions the data.
- The upper layer is scale-agnostic. It contains business logic. You write it as if scaling never happens at all. It never names a machine.
The lower layer provides a scale-agnostic programming abstraction to the upper layer. The paper’s 2007 example was Google’s MapReduce. Today you would point at a sharded datastore’s client, a service mesh, or a partitioned stream. The bet — borne out since — is that this lower layer eventually crystallizes into platforms and middleware. The messy terminal-multiplexing code of the 1970s eventually became TP-monitors, and it let programmers write only business logic.
Disjoint scopes of serializability
The second assumption is the technical heart of the paper. Helland asks you to stop imagining one global transaction scope and instead assume many disjoint scopes of serializability. Picture each machine (or tight cluster) as a separate island where ACID transactions are real and local. Between islands, there is no atomic transaction. That’s what makes them disjoint.
Each piece of data lives in exactly one island. A transaction may touch any data within a single island. It may never atomically span two islands. Replication for high availability does not change this. It only keeps copies. It does not merge the islands.
If that sounds abstract, it is exactly the world of a sharded database. A transaction inside one shard is ACID. A transaction across two shards is either impossible or expensive-and-fragile (2PC), so you avoid it.
At-least-once messaging is the water we swim in
The third assumption is about how islands talk. Consider the everyday job of consuming a message and durably writing some data:
- You receive the message (not yet acknowledged).
- You update the database.
- You acknowledge the message.
The message broker and the database are two different durable systems. The acknowledgement is not atomically coupled to the database write. Only your application code stitches them together. So there is always a failure window. Crash between the write and the ack, restart, and the message gets processed again.
The broker behaves this way on purpose. Its only alternative is to occasionally lose messages (“at-most-once”), which is worse. So the industry standard is at-least-once delivery, and the bill lands on the application developer: you must tolerate message retries and out-of-order arrival.
Entities: the unit of atomicity
Now the first big idea. If atomic transactions cannot span islands, the upper layer needs a unit it can reason about. It needs a chunk of data that it may update atomically. Helland names it the entity.
An entity is a collection of data with three properties:
- It has a unique key (an entity-key) — a customer-id, an order-id, an account number, a device serial.
- It is disjoint — its data never overlaps another entity’s data. Every datum lives in exactly one entity.
- It lives within a single scope of serializability — so you can always do an atomic transaction inside one entity, and never across two.
No rule constrains how you store an entity. Use SQL rows across several tables, a JSON document, a blob, or files — whatever fits. The primary key of those rows begins with the entity-key. An “order-processing” application is a sea of order entities. To be scalable, one order’s data must be disjoint from every other order’s.
Repartitioning means you can never assume co-location
Because the scale-aware lower layer owns placement, the location of any entity can change as the deployment evolves. Hash partitioning: two different keys land together only by luck. Range partitioning: adjacent keys usually share a machine — until one day they don’t.
This produces the nastiest class of latent bug in the paper. A test that relies on two neighboring keys being atomically updatable will pass in the test deployment, because the neighbors happened to be co-located. The bug only surfaces months later when repartitioning splits them across scopes and the atomicity silently disappears.
You can never count on different entity-key values residing in the same place.
Only one fact lets you know that you can update two pieces of data atomically: a single unique key unifies them. At that point they are just one entity. So a scale-agnostic abstraction must treat the entity as the hard boundary of atomicity.
The alternate-index tax
One consequence stings. We are used to finding a customer by id, by email, by credit-card number, by address. The database keeps all those indices perfectly consistent. Under almost-infinite scaling, the entity sits in one scope of serializability. But you must assume that the alternate index over a different key lives in a different scope.
You cannot atomically update an entity and its alternate index. The classic two-step lookup still works. Resolve the alternate key to the entity-key, then fetch the entity by entity-key. But nothing keeps the two indices transactionally in sync. What the database used to do for free now becomes the application’s job. The application maintains alternate indices asynchronously, through messaging. Readers must accept that an alternate index may be momentarily out of sync with the entity it points at.
Messaging across entities
If you cannot update two entities in one transaction, you update them in two transactions and connect them with a message. Three properties make this work.
Messages are asynchronous with respect to the sending transaction
The decision to send a message lives in the sending entity. The destination lives in another entity. By definition, you cannot update that entity atomically with the sender. So the message must not become visible at the destination until after the sender’s transaction commits.
Imagine the alternative: you send a message mid-transaction, the message escapes, and then your transaction aborts. Now the world acts on something you have no memory of causing. To prevent this, the message must be enqueued transactionally — committed atomically with the state change that decided to send it.
outbox row in the same local transaction. A separate poller relays outbox rows to the broker at-least-once. The paper described it in 2007. We gave it a catchy name later.Messages go to entity-keys, not machines
The scale-agnostic sender knows the destination’s entity-key, not its location. It is the scale-aware layer’s job to correlate the key to a physical location and deliver the message. As systems grow, the set of entities outgrows a single store. The lower layer routes by partitioning scheme, and the upper layer stays blissfully ignorant. The destination of a message is, effectively, an entity-key.
Repartitioning makes delivery messy
Entities move while messages are in flight. A message chases an entity to its old home, finds it gone, and has to follow. The clean FIFO illusion between a sender and a destination breaks: messages repeat, later messages arrive before earlier ones. This is why scale-agnostic applications converge on idempotent processing of every application-visible message, and tolerate reordering. There is no opting out.
Activities: what one entity remembers about a partner
Here is the paper’s second coinage, and the one most people miss. Every message can arrive more than once and out of order. So a recipient entity must keep state to defend itself. It organizes that state per partner. Helland calls this per-partner state an activity.
An entity that talks to many partners has many activities — one per partner entity. An activity is, plainly, “the stuff I remember about that partner.” It tracks which messages it already processed (for idempotence). It also tracks the stage that the relationship with that partner reached.
The paper’s running example: an order with many line items. There is an entity for the order and a separate entity for each item in the warehouse. And you cannot assume a transaction across them. Inside the order entity, the per-item state is an activity. That state holds the reservation protocol, the confirmed items, and the outstanding ones. This pattern already exists in large apps. It usually stays unnamed until a duplicate-message bug forces someone to build it deliberately.
Idempotence, defined carefully
Processing a message is idempotent if running it again causes no substantive change to the entity. “Substantive” is deliberately application-defined. Writing a read-audit log record is not substantive. Charging a card again is. Messages split into two buckets:
- Naturally idempotent — reads, or changes that don’t alter substantive state. Safe to replay as-is.
- Not naturally idempotent — substantive changes. Here the entity must remember it already processed this message so a retry makes no further change. And if the sender expected a reply, the entity must re-send the same reply. It cannot know whether the original sender ever received it.
That memory is state, and it accumulates in the activity, per partner. This is the conceptual parent of every processed_events / idempotency-key table you have ever written. Helland notes the obvious. You could eliminate duplicates down in the plumbing. But the dedup state would have to migrate with the entity during repartitioning, and that machinery is rarely available. So the scale-agnostic application owns idempotence itself.
Coping without atomicity: uncertainty and workflow
The final move is the deepest. Without distributed transactions, how do multiple entities agree on something — reserve inventory, confirm an order, close a deal?
With 2PC, a transaction manager holds the uncertainty of an in-flight decision in locks. Take that away and the uncertainty does not vanish. It moves into your business logic. An entity reserves inventory “without yet knowing if it will be used.” That is accepting uncertainty, in the open, as application state. Helland is blunt about what this is:
This is simply workflow. Nothing magic — just that we can’t use distributed transactions, so we use workflow.
The mechanism is the tentative operation: one entity asks another to commit-with-a-right-to-cancel. Every tentative operation eventually resolves into either a confirming operation (the entity releases the right to cancel) or a cancelling operation. Uncertainty rises as entities accept tentative operations. It falls as confirmations and cancellations arrive. That is exactly how business works, with reservations, holds, and cancellation clauses.
The escrow example
Buying a house: the buyer, seller, mortgage company, and others each enter a two-party agreement of trust with one escrow company. Nobody knows the outcome until escrow closes. Everyone accepts uncertainty meanwhile. Only the escrow company decides. It is a hub-and-spoke set of two-party relationships that makes many parties agree without a distributed transaction across all of them.
That is the scaling insight. You compose large agreements from fine-grained, two-party tentative/confirm/cancel relationships. Entity-keys link them, and activities track each partner’s known state. You do not need a global lock. You need a web of small, durable, per-partner promises.
Mapping the paper to your stack
The vocabulary is from 2007, but every term has a name you already use:
| Helland's concept | What it means | What you call it today |
|---|---|---|
| Disjoint scopes of serializability | Atomicity is local to one island | Database shards / partitions |
| Entity | Keyed, disjoint, single-scope unit of atomicity | Aggregate (DDD) · shard key · partition key |
| Activity | Per-partner state an entity remembers | Idempotency table · saga state · per-conversation state |
| Transactional enqueue | Commit the message with the state change | Transactional outbox |
| Idempotent message processing | Replays cause no substantive change | Idempotency keys · upsert / ON CONFLICT |
| Tentative / confirm / cancel | Agreement without distributed locks | Saga · compensating transaction · reserve-then-confirm |
| Alternate index in another scope | Secondary lookups can't be atomic with the entity | CQRS read model · search index lag |
The honest summary
If you remember nothing else, remember the chain of reasoning, because it’s airtight:
- At real scale you cannot rely on distributed transactions — they are too slow and too fragile.
- So atomicity stays inside disjoint islands. Each piece of data lives in exactly one.
- The unit you can update atomically is the entity, identified by a key, living in one island, never atomically joined to another.
- Entities talk only by messages, enqueued transactionally and delivered at-least-once, out of order.
- So every recipient must be idempotent, remembering per-partner state in an activity.
- And entities reach multi-entity agreement through fine-grained workflow — tentative operations that confirm or cancel — not through a global lock.
What makes the paper enduring is its modesty. Helland does not invent these patterns. He names what good engineers already did implicitly. And he argues that the names let us apply the patterns consistently, instead of rediscovering each one through a production incident. He even predicted the rest: just as the terminal-handling chaos of the 1970s hardened into TP-monitors, this hand-crafted scaling logic would harden into middleware. It did — event-streaming platforms, workflow engines, durable-execution frameworks, and CQRS toolkits are exactly that.
The goal was never exactly-once everywhere or one global transaction. It is to know precisely what guarantee you have at every boundary. Design each entity to be safe under it. And stop telling yourself the Maginot Line will hold.