There Is No Global Now — Why Distributed Systems Can't Trust the Clock, and How Lamport, Vector Clocks, and Google's TrueTime Fake It

Tags: Distributed Systems · Clocks · Ordering · Lamport Clocks · Vector Clocks · HLC · TrueTime · Spanner · Reading time: ~16 min · Category: System Design → Architecture Decisions

Ask two engineers to design a system where “the most recent write wins” and both will reach for the same thing without thinking: a timestamp. Stamp each write with the time it happened, keep the biggest one, done. It’s so obvious it feels beneath discussion. It’s also the origin of a whole family of silent data-loss bugs. It rests on an assumption that is simply false in a distributed system: that two machines can agree on what time it is. They can’t. Not approximately, not with NTP, not ever with certainty. There is no shared now. Once you internalize that, a surprising amount of distributed systems design turns out to be an elaborate campaign to work around it.

So here’s the thread the whole post hangs on. In a distributed system you almost never actually care what time an event happened — you care what order events happened in. We reach for wall-clock time because it feels like a free ordering. It isn’t free, and it isn’t an ordering. Untangling those two ideas — time versus order — is the whole journey. It takes you from “last write wins” (a bug) to Lamport clocks, vector clocks, and hybrid logical clocks. It ends at the atomic-clock trick Google uses in Spanner to buy back the illusion of a global clock.

Why the clock on the wall is lying to you

Every machine has a time-of-day clock — the thing behind System.currentTimeMillis(), Date.now(), time.time(). It reports wall-clock time. NTP (the Network Time Protocol) synchronizes it to the outside world. NTP asks a set of time servers “what time is it?” and moves the local clock toward the answer. This sounds authoritative. It is not, and the reasons accumulate:

• NTP is only as good as the network. It estimates the correct time by measuring round-trip delay and assuming the path is symmetric. On a congested or asymmetric link that assumption breaks. Public NTP sync realistically keeps you within tens of milliseconds of the correct time — sometimes worse. Milliseconds are an eternity when writes are landing microseconds apart.
• Clocks drift. A quartz oscillator runs slightly fast or slow depending on temperature and age. Between NTP corrections, a machine’s clock wanders on its own. Different machines wander in different directions.
• Clocks jump — and run backwards. When NTP decides the local clock is ahead, it can yank it backwards. Your code can call Date.now() twice and get a smaller number the second time. Any logic that assumed time only moves forward is now wrong.
• Leap seconds. Occasionally a minute has 61 seconds. Systems that never accounted for it broke Reddit, Cloudflare, and others when the extra second arrived.
• The machine pauses. A GC stop-the-world, a VM live-migration, an overcommitted hypervisor, or a laptop lid closing can freeze a process for hundreds of milliseconds or many seconds. When it resumes, wall-clock time is far ahead. The process never knew it was gone.

Monotonic clocks fix half of it (and only half)

There’s a second clock in every machine that people forget about: the monotonic clock (System.nanoTime(), clock_gettime(CLOCK_MONOTONIC), performance.now()). It only ever moves forward, at a steady rate. It is immune to NTP jumps and leap seconds. If you want to measure how long something took — a request latency, a timeout — this is the clock to use. Wall-clock time for durations is a classic bug. A mid-measurement NTP correction can give you a negative elapsed time.

But the monotonic clock has no meaning across machines. Machine A’s nanoTime() and machine B’s count from two unrelated, arbitrary starting points. It answers “how long on this box?” beautifully and “which box’s event came first?” not at all. So it fixes durations and does nothing for our real problem — ordering events that happened on different machines.

The bug this actually causes: last-write-wins

Let’s make the failure concrete, because it’s not hypothetical — it’s the default conflict-resolution strategy in more databases than you’d like. Last-write-wins (LWW): when two writes touch the same key, keep the one with the larger timestamp, discard the other.

Two clients update the same shopping-cart key at nearly the same instant. Node A handles client 1’s write, and node A’s clock runs a hair fast. Client 2’s write genuinely happened after client 1’s. Node B handles it, and node B’s clock runs a hair slow. Now compare timestamps:

Nothing errored. No log line fired. The database did exactly what it promised. It kept the larger timestamp. In doing so it deleted a write that really happened, because it trusted two clocks it had no right to trust. This is the quiet violence of LWW: it doesn’t fail, it forgets.

The reframe: you wanted order, not time

Step back and notice what LWW was really trying to do. It didn’t care that write 2 happened at 11:59:59.988. It cared that write 2 happened after write 1. Time was just a convenient proxy for order — and a broken one. So drop time and ask the sharper question directly: given two events, can we tell which one happened before the other?

In 1978 Leslie Lamport answered this in one of the most cited papers in computer science. His answer defines “before” without any clock at all. Event a happens-before event b (written a → b) if any of these hold:

• Same process: a and b happen on the same node and a comes first in that node’s local sequence.
• Message passing: a is the sending of a message and b is the receipt of that same message. You cannot receive a message before the sender sends it.
• Transitivity: if a → b and b → c, then a → c.

The profound part is what’s missing. If neither a → b nor b → a holds, the events are concurrent. And “concurrent” does not mean “at the same time.” It means causally independent. Nothing connects them, so asking which came first is meaningless. Two writes on two disconnected nodes are concurrent. That holds even if one is an hour after the other on a god’s-eye clock. Neither one could influence the other.

Lamport clocks: cheap order from a single counter

Lamport turned happens-before into something you can actually compute with one integer per node. The rules fit on a napkin:

• Each node keeps a counter L, starting at 0.
• Before any local event (including sending a message), increment: L = L + 1.
• Every message you send carries the sender’s current L.
• On receiving a message with timestamp Lmsg, set L = max(L, Lmsg) + 1.

// A Lamport clock is one integer and two rules.
let L = 0;

function localEvent() {
  L = L + 1;                 // tick before doing work
  return L;
}

function send(msg) {
  L = L + 1;
  msg.ts = L;                // stamp the outgoing message
  network.send(msg);
}

function receive(msg) {
  L = Math.max(L, msg.ts) + 1;   // absorb the sender's knowledge, then tick
  handle(msg);
}

That max on receive is the whole trick. It drags the receiver’s clock forward to at least the sender’s, so causality flows through messages. The guarantee you get is precise:

And here is the guarantee you don’t get. It confuses everyone the first time. The implication only runs one way. L(a) < L(b) does not mean a → b. Two concurrent events can have any numbers at all — equal, or one bigger. The numbers only record how many local ticks each node did. So a Lamport clock can respect causality but can’t detect it. Given two timestamps, you cannot tell whether one truly caused the other or they were independent.

Even so, Lamport clocks are enormously useful. A total order is a single line everyone agrees on, and a replicated log needs one. To get it, you break ties on the counter with the node ID: order by (L, nodeId). Now every pair of events has a definite, deterministic order that never contradicts causality. That’s exactly what you need to send writes through a leader and have every replica apply them identically. This idea therefore sits underneath much of consensus and replication.

Vector clocks: when you must detect concurrency

Sometimes “respect causality” isn’t enough and you need to detect it. You must look at two writes and know one of two things for certain. Either one happened before the other, or they were truly concurrent (a real conflict). That’s the job of the vector clock. Instead of one integer, each node carries a whole vector: one counter per node in the system.

• Node i keeps a vector V of length N, all zeros.
• On a local event, node i increments its own slot: V[i] += 1.
• Messages carry the full vector.
• On receive, take the element-wise max of your vector and the incoming one, then increment your own slot.

Now comparison becomes exact. To compare two vectors, compare them slot by slot:

• V(a) < V(b) in every slot (and they’re not equal) ⇒ a → b. a definitely happened before b.
• V(a) > V(b) everywhere ⇒ b → a.
• Neither dominates — a is bigger in some slot, b in another ⇒ concurrent. A genuine conflict you must resolve, not silently drop.

This is how Amazon Dynamo and its descendants (Riak, Voldemort) chose to handle conflicts. They do not let a clock secretly pick a winner. They detect the conflict instead. Then they either merge it automatically or give both siblings back to the application to reconcile. An automatic merge takes the union of both versions, which is the famous “resurrected deleted item” trade. The cost is obvious in the shape of the data structure. A vector clock is O(N) in the number of nodes that have ever written. So on a large, churny cluster the vectors themselves grow and need pruning. You buy exact concurrency detection and pay for it in metadata.

Hybrid Logical Clocks: physical time you can actually order by

Here’s the tension. Logical clocks order events correctly, but they have no link to real time. Physical clocks track real time but order events incorrectly. Hybrid Logical Clocks (HLC) — Kulkarni et al., 2014 — ask the natural question. Can we glue a physical timestamp and a logical counter together and get most of both?

An HLC timestamp is a pair. The physical part pt is roughly wall-clock milliseconds. The small logical counter c only advances when the physical clock hasn’t moved. On every event, the clock takes the max of its own physical time and the incoming timestamp’s physical time. If they’re equal, it increments the logical counter to break the tie. The result stays within a tiny bounded distance of true wall-clock time, and it never violates happens-before. The logical counter absorbs a stalled or backwards-jumping physical clock instead of corrupting the order.

TrueTime: what if you just bought better clocks?

Everything so far accepts that clocks are untrustworthy and designs around the problem in software. Google, building Spanner, asked the opposite question. What if we make the clocks trustworthy enough to use directly? And, crucially, what if each machine can know how wrong its own clock might be?

Spanner’s TrueTime is a clock API backed by GPS receivers and atomic clocks in every datacenter. Its twist isn’t that it returns a timestamp. It returns an interval instead: [earliest, latest]. TrueTime guarantees that the true current time lies somewhere inside it. Instead of pretending it knows the exact time, it gives you an honest error bar, typically just a few milliseconds wide. That single change — exposing the uncertainty instead of hiding it — is the whole idea.

// TrueTime doesn't return "now". It returns an interval that
// is GUARANTEED to contain the true "now".
const tt = TrueTime.now();   // { earliest, latest }
const uncertainty = tt.latest - tt.earliest;   // usually a few ms

// To guarantee a commit at time T is really in the past
// everywhere before anyone can read it, Spanner just... waits
// out the uncertainty. This is "commit-wait".
function commit(txn) {
  const T = TrueTime.now().latest;   // pick the latest possible instant
  writeCommitRecord(txn, T);
  waitUntil(TrueTime.now().earliest > T);  // sleep past the error bar
  release(txn);
}

With an honest interval in hand, Spanner does something clever called commit-wait. The transaction picks a commit timestamp. Then it deliberately sleeps until TrueTime guarantees that instant is now unambiguously in the past. The error bars no longer overlap. Once it wakes, Spanner can no longer give any transaction anywhere an earlier timestamp than one it should have come after. Spanner literally spends a few milliseconds of latency to buy back the global ordering that clock uncertainty stole. Tighter clocks (smaller error bar) mean shorter waits — so Google’s investment in atomic clocks is, very directly, an investment in lower transaction latency.

The four approaches, side by side

Where you actually meet this

You rarely implement a clock scheme from scratch. You inherit its consequences from a system you run, and the bugs appear as “impossible” data:

• Cassandra / ScyllaDB. LWW on wall-clock timestamps. If your app nodes’ clocks drift, concurrent writes to the same cell get silently lost. This is why “keep NTP tight” is not optional operational advice there — it’s a correctness dependency.
• DynamoDB / Riak. Vector-clock lineage (or version vectors) surface conflicts as siblings your app must merge. If you’ve ever wondered why a deleted item “came back,” that’s conflict resolution choosing availability over your intent.
• CockroachDB / YugabyteDB. HLCs order distributed transactions. They also assume a maximum clock offset — exceed it and a node removes itself from the cluster rather than risk serving stale reads. Clock skew is a first-class failure mode, monitored and enforced.
• Kafka. Message timestamps are wall-clock and can go backwards across producers. Order within a partition comes from offsets (a Lamport-like monotonic counter), not from timestamps. Sorting a topic by timestamp is a bug waiting to happen.
• Spanner / Cloud Firestore. TrueTime + commit-wait give externally consistent global transactions — the gold standard, paid for in hardware and a few milliseconds of commit latency.

The takeaway

“Stamp it with the current time and keep the latest” is a fine mental model. It holds right up until two machines have to agree on what “latest” means. Then it quietly deletes your data. The fix is a single reframe, and everything else follows from it:

1. You want order, not time. Wall-clock time is a broken proxy for ordering. Ask “what happened before what?” That’s causality, and happens-before defines it without any clock. Events that aren’t causally related are concurrent, and inventing an order for them is inventing data.
2. Pick the clock that matches the question. Need a cheap total order for a log? Lamport. Need to detect real conflicts? Vector clocks. Need ordering that also reads like real time? HLC. Need externally-consistent global transactions and can spend hardware? TrueTime — whose real lesson is to measure your uncertainty and wait it out rather than pretend it’s zero.

So the next time you reach for a timestamp to decide who wins, stop and ask two questions. They separate a correct system from a silent data-loss machine. Do these two events actually have a causal order at all? And am I comparing two facts, or two error bars I’m pretending are points? In a distributed system there is no global now — the good designs are the ones that stopped pretending there was.