Full lesson
Explore the full explanation, examples, and visuals at your own pace.
Agent pattern
General knowledgeWhere should your Collector actually run? In the agent pattern, right next to the app. Checkout's SDK sends its telemetry over OTLP, the OpenTelemetry Protocol, to a Collector on the same host, like a sidecar or DaemonSet. That Collector then exports everything on to the backend.
Gateway pattern: one shared endpoint
The gateway pattern flips that. Instead of a Collector beside every app, Checkout sends OTLP to one shared endpoint, often one per cluster or region. Behind it sits a load balancer, which just spreads requests. It fans the data out to Collector 1 and Collector 2, and both forward to the backend.
What each pattern costs you
Each pattern has a price. Agents are easy to start, with one Collector per app, but they scale poorly. Gateways centralize credentials, filtering and sampling, but you pay with another thing that can fail, extra latency, and higher resource costs.
- Agent: simple, one Collector per app
- Gateway: central credentials and policy
- Gateway cost: extra hop, more resources
Plain load balancing splits trace abc123
General knowledgeTail sampling decides what to keep only after it has seen the whole trace. Now look at the flow. Checkout sends a span for trace abc123 to the load balancer, which forwards it to Collector 1. Payment sends its span for the same trace, and this time it lands on Collector 2. The trace is split, because a plain load balancer ignores trace IDs.
Two tiers: route by trace ID
Add a first tier, and the problem goes away. Checkout and Payment both send their abc123 spans to a Tier 1 Collector running the load-balancing exporter, keyed on traceID. Every span with that trace ID goes to Collector 1. So Collector 1 holds the whole trace, and tail sampling can finally judge it.
Single-writer principle
Routing solves one problem, but gateways create another. Every metric stream needs a single writer and a unique identity. Here, Collector 1 and Collector 2 both write Metric M1 for the same unidentified resource. Prometheus sees one series, so the two sources can overwrite each other and corrupt the data.
Symptom: time runs backwards
Watch the arrival order. Metric M1 shows 100 at 13:56:04, then 120 at 13:56:24, then 110 stamped 13:56:04 again. Time runs backwards, and Prometheus reports out-of-order samples. The fix is unique identity, using the Kubernetes attributes processor or the resource detector processor.
Chart values
| Arrival order | Value |
|---|---|
| T1 13:56:04 | 100 |
| T2 13:56:24 | 120 |
| T3 13:56:04 | 110 |
Takeaways
An agent Collector sits beside the app and keeps things simple. A gateway adds central control, at the cost of extra hops and resources. Start with agents, then add a traceID-routed gateway when you need tail sampling or central policy.
- Agent: beside the app, simple, limited scaling
- Gateway: central control, more hops and cost
- Route by traceID; one writer per metric

