Averages Are a Lie — The Long Tail That Quietly Runs Every System You Build, From p99 Latency to Hot Keys

Tags: Long Tail · Power Law · Tail Latency · p99 · Zipf's Law · Caching · Hot Keys · Percentiles · Reading time: ~16 min · Category: System Design → Observability

Somewhere in school a teacher handed you the bell curve. It quietly became your default picture of the world. Most things cluster near the average, extremes are rare, and the mean is a fair summary of the whole. It’s a beautiful model. It’s also almost never the shape of the data a software engineer actually deals with. Request latencies, key popularity, file sizes, follower counts, query frequencies — these don’t hump around a mean. They plunge from a tall head into a long, thin tail that stretches on and on without ever touching zero. And once you see that shape, a startling number of “weird” production problems stop being weird. They were the tail all along.

Here’s the thread the whole post hangs on: in software, the average is a lie, and the tail is where everything expensive lives. The same long-tail shape appears in two places you fight every week. One is the distribution of your response times, where a few requests are horrifically slow. The other is the distribution of your traffic, where a few keys are white-hot and most are ice-cold. They look like unrelated problems. They’re the same distribution wearing two costumes, and learning to recognize it changes how you measure, cache, and scale everything.

Why the bell curve betrays you

A normal distribution has a characteristic scale. Human height is normal. The average is ~170 cm, and you will never, ever meet someone 10× taller. The mean tells you almost everything, and outliers are genuinely negligible.

Software data is the opposite. It is heavy-tailed. Often it follows a power law, where the frequency of an item falls as a power of its rank. The signature of a power law is scale-invariance. There is no “typical” value, and outliers are not negligible. They dominate. A handful of items can carry more weight than the entire rest of the distribution combined. The famous shorthand is the 80/20 rule, the Pareto principle. 20% of the causes drive 80% of the effects. In real systems the split is often more brutal — 1% of keys serve 90% of reads.

Costume #1: the latency tail — your p99 is the real story

Plot the response times of any busy service and you get the long-tail shape. You see a tall spike of fast responses and a thin tail of slow ones that drags far to the right. This is why serious teams never speak of “average latency.” They speak in percentiles:

• p50 (median) — half your requests are faster than this. The typical experience.
• p99 — 99% are faster. The slowest 1% are worse. This is the tail.
• p99.9 — the slowest 1 in 1,000. Rare per request — but if a user makes 1,000 requests in a session, they will almost certainly hit it.

The gap between p50 and p99 is often 10–100×. A service can have a delightful 5 ms median and a 900 ms p99. The average then reports a number in between that is true of nobody. The people in your p99 are frequently your most engaged users. They have the biggest carts, the most data and the fullest feeds. Bigger accounts take slower paths. The tail isn’t your worst customers. It’s often your best ones.

The multiplier nobody expects: tail amplification

This is where the tail stops being a monitoring footnote. It becomes a force that defines your architecture. In a modern system, one user request rarely reaches only one server — it fans out to dozens or hundreds. Loading a single feed might touch 100 backend services in parallel. The parent request cannot finish until the slowest child returns. So the user experiences the maximum of 100 latencies, not the average.

In “The Tail at Scale,” Jeff Dean and Luiz Barroso made this math famous. Now do it yourself. Suppose each backend has a perfectly respectable p99 — only 1% of its responses are slow. If a request fans out to N independent servers, the probability that at least one is slow is:

// Probability the whole fan-out request hits a slow leaf.
function chanceOfSlow(p, N) {
  return 1 - Math.pow(1 - p, N);   // 1 - P(all fast)
}

chanceOfSlow(0.01, 1);    // 0.01   → 1% of requests slow
chanceOfSlow(0.01, 100);  // 0.633  → 63% of requests slow  😱
chanceOfSlow(0.01, 200);  // 0.866  → 87% of requests slow

What you do about the latency tail

You cannot delete the tail. GC pauses, queueing, cold caches, a noisy neighbor on the box, and the occasional retstorm will always produce slow responses. So the techniques do not prevent tail events. They stop a slow server from making the user wait:

• Hedged requests. Send the request to one replica. If it has not answered by the 95th-percentile mark, send a duplicate to a second replica. Take whichever replica returns first. You spend a few percent extra load to chop the tail dramatically. Two independent draws rarely both land in the tail.
• Tied requests. Same idea, tighter. Dispatch to two replicas that tell each other the moment one starts executing, so the other cancels. Kills the tail with almost no wasted work.
• Aggressive timeouts + budgeted retries. Abandon a straggler fast and continue. But cap retries and use a retry budget. If you do not, you turn a tail event into a self-inflicted overload. The backpressure story warns about that exact failure.
• Trim the fan-out. Fewer leaves per request means a smaller N in that 1-(1-p)^N. Sometimes the cheapest tail fix is to ask fewer services.

Costume #2: the popularity tail — a few hot keys, a sea of cold ones

Now the same shape in your traffic. Rank your keys — products, videos, users, URLs — by request volume and you get Zipf’s law. In that law, the frequency of the k-th most popular item is roughly proportional to 1/k. The #1 item gets twice the traffic of #2, three times #3, and so on. The result is a tiny, screaming-hot head and an enormous, whisper-quiet tail.

This one distribution explains a remarkable amount of your architecture:

• Why caching works at all. Because the head is so concentrated, a cache that holds the top few percent of keys can serve the large majority of reads. An 80/20 (or 90/10) split is why a small, cheap cache in front of a big, slow database works so well.
• Why caching cannot save the tail. The long tail is, by definition, requests for things almost nobody asks for. So they are almost always cache misses. Every tail request pays full price at the origin. Your cache hit rate can be 95%, and the 5% of misses are the tail. Each one is slow. That is exactly the part a cache cannot help.
• The hot-key / hot-partition problem. The head is too concentrated. When one key is white-hot — a viral tweet, a flash-sale SKU, a celebrity’s profile — it lands on a single shard. It melts that shard while every other shard sits idle. Consistent hashing spreads keys evenly by count. But the long tail makes keys very uneven by traffic. So “balanced” shards can carry radically unbalanced load. (This is the shadow side of consistent hashing.)

The business long tail (and why it’s an engineering bill)

Chris Anderson’s 2006 book made the phrase “long tail” popular. Physical stores can only stock the head, because shelf space is scarce. Online stores have near-zero shelf cost, so they can sell the entire tail at a profit. That tail is millions of niche books, songs and products, and each one sells a few copies. Amazon, Netflix and Spotify are long-tail businesses. Together, their obscure catalog rivals the hits.

That business win lands on your desk as an engineering cost, because the tail has the opposite economics of the head. Head items are cache-hot, CDN-resident, and cheap per request. Tail items are cache-cold, disk-resident, and expensive per request — every access is a miss that reads from cold storage. Serving the tail is where storage tiering comes from. Keep the hot head on fast SSD/RAM. Move the cold tail to cheap object storage (S3, glacier-class). Accept higher latency for the rare hits. You literally shape your infrastructure to match the two halves of the distribution.

Where you actually meet the long tail

• SLOs and dashboards. Mature teams write every SLO on percentiles (“p99 < 200 ms”), never on averages. The average hides the tail that actually breaks users. If your dashboard shows mean latency, it lies to you.
• Google / big-fan-out search. “The Tail at Scale” exists because the slowest shard defines a query that touches thousands of leaf servers. Google engineers invented hedged and tied requests to tame exactly this.
• CDNs & caches. Hit ratio follows Zipf. The head caches well. The tail is your origin-load floor, and you cannot cache it away.
• Redis / DynamoDB hot partitions. The classic incident is “one key is 1000× the others” — flash sales, viral content. You solve it with replication or key-splitting.
• Search & recommendations. Most distinct search queries are rare (the tail), so you cannot precompute them all. Head queries get cached results. The service computes tail queries live.
• Kafka / partitioned pipelines. A skewed partition key (one customer 100× the rest) creates a hot partition. The popularity tail appears as a lagging consumer.

The takeaway

The bell curve taught you that the average is a fair summary and outliers are rare. In software, both halves of that sentence are false. The data is heavy-tailed, and the outliers are the story. One reframe carries the whole idea:

1. Stop trusting the average. Measure the tail. Response times and popularity are long-tailed, so the mean describes no one. Speak in percentiles (p50/p99/p99.9). Remember that fan-out amplifies the tail. A leaf’s p99 becomes a parent’s median. You do not remove the tail. You stop it from reaching the user (hedged/tied requests, tight timeouts, smaller fan-out).
2. Design the head and the tail as two different systems. The hot head is cacheable, cheap, and at risk of hot-partition meltdown — spread it. The cold tail is uncacheable, expensive per hit, and vast in aggregate. Tier it to cheap storage and accept graceful slowness. One size fits neither.

Sometimes a metric looks fine but users are complaining. Sometimes a “balanced” cluster has one node on fire. So next time, ask the two questions the long tail always poses: am I looking at an average that’s hiding a tail? and is this the hot head or the cold tail — because they need opposite designs. The engineers who ship reliable systems aren’t the ones who optimized the average. They’re the ones who went looking for the tail before it found them.