Most of your traces look exactly alike. In this illustrative checkout system, about 1,980 traces per second are healthy, and only around 20 are errors or slow. Keeping everything gets expensive, so the real question is which traces to keep, and when to decide.

Before we go further, let's fix two terms. A sampled trace is one the sampler chose as a representative, so it's processed and exported. A not sampled trace isn't. That's why saying you "sampled out" data, or that dropped data was sampled, is incorrect.

A cook doesn't need to drink the whole pot. One spoonful from a well-stirred pot tells you how it tastes. That's representativeness: a smaller group accurately stands in for a larger one. Unlike filtering or aggregating, you can verify it mathematically. And in high-volume systems, 1% or less can often be enough.

Sampling fits when you produce around a thousand traces per second, most of it healthy and similar, and you can state clear rules, like keep errors or slow requests. It also helps to tell services apart and route unsampled data to cheap storage.

Sampling isn't always the right call. If you only produce tens of small traces per second, there's little to save. If you only use aggregate data, you can pre-aggregate instead. And if regulation forbids dropping data, and you can't route it to cheap storage, don't sample.

Sampling isn't free. You pay for compute, like a tail sampling proxy. You pay engineering time to keep rules current. And poor sampling can drop critical data. Sometimes, simply paying for more observability capacity turns out cheaper.

Head sampling decides at the start. Each checkout request gets a trace ID, a unique identifier shared by every span in that trace. The sampler checks that ID against a 5% target, then keeps or drops. Because the ID is the same throughout, whole traces are kept, with no missing spans.

Follow one checkout request. Checkout asks the Head Sampler about the trace ID, and the answer comes back: not sampled. Only then does Checkout call Payment to charge the card, and Payment returns an error. But the decision is already made, so that error is never exported. With a 5% rate, this error trace has a 95% chance of being lost. Head sampling alone can't guarantee error traces.

Tail sampling flips the timing. Spans are buffered in the tail sampler until all or most of the trace has arrived. Then the policy check looks at the whole trace, sees the Payment error, and exports it. Traces with no match are dropped. Tail sampling decides after it has seen the trace.

Because tail sampling sees the whole trace, your rules can get specific. Keep every error, keep slow traces, favor a newly deployed service, or sample low-volume and high-volume services at different rates. That's why larger systems almost always end up needing it.

Both aim to keep the traces worth keeping, but they pay differently. Head sampling is a light gate: easy, efficient, and it works anywhere in the pipeline, yet it can't see content. Tail sampling is the warehouse: stateful, sometimes needing many nodes, with evolving rules, and often tied to a vendor.

You can use both. Services send high-volume traffic to the Collector's Probabilistic Sampling processor, which keeps only a small share. Then the Tail Sampling processor picks the interesting traces, like errors, and sends those chosen traces to the backend. Head sampling protects the pipeline from overload. Many vendors offer similar solutions too.

Head decides before the trace happens; tail decides after seeing it. So if you must catch every error trace, you need tail sampling.