Full lesson
Explore the full explanation, examples, and visuals at your own pace.
Illustrative: 1000 traces per second
In a high-volume system, keeping just 1% of traces can still tell you the truth. Here, a thousand traces per second shrink to ten. That works because of representativeness: a small group can accurately stand in for a larger one. You keep the insight, and cut the cost.
Chart values
| Label | Traces per second |
|---|---|
| Generated | 1000 |
| Sampled | 10 |
Say it the right way
A sampler is the component that chooses which traces to keep. A sampled trace is chosen, then processed and exported. A not sampled trace isn't. So dropped data is never sampled, and saying "sampling out" data is incorrect.
- Sampled: processed and exported
- Not sampled: not processed or exported
- Dropped data is never sampled
Should you sample at all?
Sampling pays off around a thousand or more traces per second, mostly healthy. With only tens of traces, or rules against dropping data, skip it. Sampling also has costs: compute, engineering upkeep, and the risk of missing critical data.
- Sample: 1000+ traces/sec, mostly healthy
- Skip: tens of traces per second
- Skip: rules forbid dropping data
Head sampling: decide at the start
A checkout request comes in and starts a trace, which gets a trace ID. The head sampler checks that ID against a 5% rate, and decides right away: sampled or not sampled. Because the ID drives the choice, whole traces are kept consistently. It's easy to configure, cheap, and efficient.
The blind spot: errors happen later
Watch the order of events. Checkout API starts the trace, and the head sampler answers right away: not sampled. Only after that does Checkout API call Payment to charge the card, and Payment returns error 500.
The decision was already made, so the error never gets exported. With a 95% drop chance, this trace most likely vanishes. Head sampling alone can't guarantee keeping error traces.
Tail sampling: decide after the trace completes
Tail sampling waits. Both the Checkout span and the Payment span are buffered in the tail sampler, which sees Payment's error and sends the trace to the backend. Healthy traces that aren't chosen are not sampled. Latency, attributes, or per-service rates work too. Head sampling decides at the door; tail sampling decides after everything.
The trade-off
Head sampling is a small, cheap gate. It's easy to configure and can run at many points in the pipeline, but it can't see later errors. Tail sampling gives you smart rules, but it needs stateful storage, sometimes many nodes, monitoring, evolving rules, and is often vendor-specific.
Combine both for very high volume
For very high volume, you can use both. All traces flow from the services into the head sampler, which keeps only a small share. That protects the pipeline from overload. Then the tail sampler applies whole-trace rules, like keeping errors, and only those kept traces reach the backend.
Head decides at the door; tail decides after seeing everything.
Head decides at the door; tail decides after seeing everything. Start with head sampling for simplicity, then add tail sampling when you must keep errors or slow traces.


