If a tree falls in a forest and no one hears it, did it make a sound? Distributed systems face a crueler version every second. A server stops answering you — did it die, or did your message just get lost on the way? You can’t tell. Not “it’s hard” — you provably cannot tell from where you’re standing.
So here is the one truth this whole post depends on. “I can’t reach it” is not the same as “it’s dead.” One is about the network. The other is about the node. Treating them as the same is how a healthy cluster talks itself into a failover at 3 AM. Every system you’ve shipped has a tiny subsystem that makes this call. The naive version, “send a ping, no reply in a few seconds, mark it dead,” is wrong in a few interesting ways. Two words fix it: suspect and corroborate.
The lie in “no reply means dead”
Here’s the detector almost everyone writes first:
async function isAlive(peer) {
try {
await ping(peer, { timeoutMs: 3000 });
return true;
} catch {
return false; // <-- "no reply in 3s, therefore dead"
}
}
The bug isn’t the code — it’s the word therefore. A missing reply has at least four explanations, and you can’t tell them apart:
- Dead — the process crashed and will never answer.
- Slow — it’s alive but mid-GC-pause or CPU-starved. The reply is coming.
- Unreachable from you — a partition ate your packet. Other nodes still reach it fine.
- Ack lost — it answered. The answer never reached you.
That’s why the literature is picky about words. A process that truly stopped is dead. One you only think stopped is suspected. A failure detector never gives you the first — only ever the second.
This is just FLP impossibility in work clothes. In an asynchronous system you can’t reliably tell a crashed node from a slow one. So no protocol can guarantee consensus. The practical fix is to accept a detector that’s allowed to be wrong. That is enough to build real systems on.
The trade-off you can’t escape
Judge any detector on two things in tension: completeness (every dead node is eventually suspected) and accuracy (live nodes aren’t falsely accused). The provable catch: you can’t be both perfectly fast and perfectly accurate. Tight timeouts catch crashes quickly, but they accuse nodes that merely paused. Loose timeouts are accurate but slow to react.
The five families of failure detector
Everything past the naive detector is a variation on one of five ideas. The five climb a ladder: one node’s lonely guess, asking others to corroborate, quantifying the doubt, pooling the whole cluster’s view, weaponizing silence itself. For each, the same three questions — how it works, what it buys you, and what it costs.
1. Ping / heartbeat + timeout
Two ways to check a peer, differing only in direction. A ping: you ask “are you there?” and wait for an ack. A heartbeat: the peer keeps sending “still here,” and you watch for its absence. Both land in the same place — a table of peers with a last-seen time. The watcher suspects anyone too quiet. Akka’s deadline detector is the textbook example.
The catch with this whole family: it captures one node’s point of view. P1 deciding P2 is dead says nothing about whether P3 can still reach P2. Fixing that blind spot is what everything below is about.
- Dead simple — one message and a timer.
- Minimal overhead — nothing to tune but the timeout.
- Perfectly adequate on stable, single-datacenter links.
- Single vantage point — can’t tell “dead” from “unreachable from me.”
- One fixed timeout for all weather — the fast⟷accurate slider nailed down.
- Trips on GC pauses and transient slowness (false positives).
2. SWIM — don’t take your own word for it
The single best idea to steal. When your direct ping to a peer goes unanswered, don’t conclude anything — outsource the question:
- P1 pings P2. No reply.
- P1 picks a few random members (P3, P4) and asks each: “ping P2 for me.”
- If any of them gets a reply, it relays the ack back to P1.
- Only if every indirect probe also fails does P1 suspect P2.
Now several nodes judge P2 from their own vantage points. A broken P1→P2 link stops masquerading as a dead P2. The probes also run in parallel, so the answer is fast. Each node only needs to know a subset of peers, which is why it scales. Serf and Consul build their membership on this.
- Corroborates before accusing — kills false positives from one-sided partitions.
- Indirect probes fan out in parallel, so it stays fast.
- Scales — each node tracks only a subset of peers.
- More messages than a bare ping.
- Needs healthy helper nodes to relay the probe.
- Still a binary up/down verdict — no confidence level.
3. Phi-accrual — “alive” isn’t a boolean
Every detector so far ends in up/down. The phi-accrual (φ) detector (used by Cassandra and Akka) outputs a number instead: a suspicion level that rises the longer a node is silent. It keeps a sliding window of recent heartbeat gaps. It learns what “normal” spacing looks like. Then it asks how surprising the current silence is, given that history. When φ crosses a threshold, the detector suspects the node.
The win is self-tuning. On a tight datacenter link, a 2-second gap is shocking and φ spikes at once. On a flaky cross-region link where arrivals already scatter, the same gap is unremarkable and φ barely moves. You stop hand-picking one timeout for all weather. You pick a confidence level instead, and the math adapts the effective timeout to conditions.
- Emits a continuous suspicion level, not a boolean.
- Self-tunes to the link — you set a confidence, not a raw timeout.
- Adapts automatically to datacenter vs cross-region latency.
- Needs a sliding history window — memory plus a warm-up period.
- Assumes inter-arrival times are roughly modelable.
- Still one observer’s view unless paired with corroboration.
4. Gossip — aggregate the whole cluster’s view
SWIM gets a few extra opinions. Gossip-style detection makes everyone’s view of everyone available without anyone broadcasting to all. Each node keeps a table of heartbeat counters. Periodically it bumps its own counter and sends the table to one random peer, who merges in the higher counters. Counters spread hop by hop like an infection. If a node’s counter hasn’t advanced anywhere for long enough, the cluster declares it failed.
The payoff is reliability through redundancy. One broken link can’t fake a death, because the counter still arrives by another route. The cluster’s view is also an aggregate that’s hard to fool. The cost is more messages. But gossip keeps the bandwidth per node bounded, and that bandwidth grows only linearly with cluster size.
- Aggregates everyone’s view — one broken link can’t fake a death.
- Redundant paths route around partitions.
- Bounded per-node bandwidth — scales roughly linearly.
- Detection latency grows with gossip rounds (hop-by-hop spread).
- Higher total message volume than a direct check.
- Larger membership state carried on every node.
5. FUSE — propagate failure with silence
One neat inversion. Instead of sending news of a failure, FUSE spreads it by stopping. Group the processes. Everyone pings everyone. The rule: the moment a process notices a peer go silent, it stops responding too. Silence is contagious. Non-response cascades like dominoes until the whole group goes quiet. By then every member knows the group failed. It’s guaranteed and cheap, since silence can’t itself be partitioned away. The trade-off is bluntness: one cut-off node can escalate to a whole-group failure — sometimes a bug, sometimes exactly the fan-out you want.
- Guaranteed and cheap — silence can’t be partitioned away.
- The whole group learns of failure with no explicit broadcast.
- Ideal when you want all-or-nothing group failure.
- Blunt — one cut-off node can collapse the entire group.
- Can’t isolate a single failure (“just P2 is down”).
- Escalates rather than contains.
The five families, side by side
| # | Approach | Core idea | False-positive resistance | Where you’ve met it |
|---|---|---|---|---|
| 1 | Ping / heartbeat + timeout | One node’s up/down verdict | Low | Akka deadline detector · k8s liveness probes · LB health checks |
| 2 | SWIM (indirect probes) | Get a second opinion before accusing | Medium–High | Serf · Consul membership |
| 3 | Phi-accrual (φ) | Continuous, self-tuning suspicion level | Medium (adaptive) | Cassandra · Akka cluster |
| 4 | Gossip heartbeat table | Aggregate cluster-wide view | High | Dynamo-style rings |
| 5 | FUSE / quiescence | Propagate failure by going silent | n/a (all-or-nothing) | Group-failure escalation |
The takeaway
“Send a ping, no reply, mark it dead” isn’t a failure detector. It’s the cheapest one, pinned to the most trigger-happy corner of the curve. Exactly one opinion stands behind it. It works until the day a GC pause or a one-sided partition talks your cluster into a failover. Two truths replace it:
- You can only suspect, never know. So aim for an honest, tunable confidence level — which is exactly what phi-accrual makes explicit.
- One node’s view is the weakest evidence. So corroborate — outsource the probe (SWIM), aggregate the cluster (gossip), or let silence carry it (FUSE).
Then pick your spot deliberately. Cheap false positive (drop a cache replica, retry elsewhere)? Lean fast. Expensive one (fence a primary, trigger an election)? Lean accurate and demand corroboration. The subsystem that quietly decides who’s alive deserves as much care as the consensus protocol on top of it. That protocol is only ever as good as the suspicions you feed it.