Full lesson
Explore the full explanation, examples, and visuals at your own pace.
Part 1
The Backend is restarting, so the Gateway can't deliver its spans. Normally your app sends spans to the Agent, the Agent forwards them to the Gateway, and the Gateway sends them on to the Backend. That last hop is failing right now. So where do the spans wait?
Step 1: buffer in memory
The new piece is the Memory Queue. The Gateway has an exporter, the part that sends data out, and the exporter has a sending queue, its waiting line. When the Backend is healthy, batches pass straight through. During an outage, they wait in memory and the exporter retries, instead of dropping them.
Part 3
General knowledgeThe Gateway sends a batch, and the Backend says it's unavailable. So the Gateway waits and retries. It fails again, so it waits longer. That's exponential backoff, waits that keep growing, with jitter adding small random offsets so retries don't all land at once. By default, retries last up to 5 minutes. Here, the Backend returns in time, and the batch is accepted.
Illustrative fill rate; default capacity about 1000 batches
In this illustrative outage, the Gateway's queue climbs from zero to about a thousand batches by minute four, then flat-lines. That's the cap. Once it's full, new data is dropped. Separately, past the five-minute retry limit, the oldest data is dropped. You can tune both limits.
Chart values
| Outage time | Batches queued |
|---|---|
| 0 min | 0 |
| 1 min | 300 |
| 2 min | 600 |
| 3 min | 900 |
| 4 min | 1000 |
| 5 min | 1000 |
Step 2: survive a Gateway crash
A memory queue disappears if the Gateway crashes. A write-ahead log, or WAL, fixes that by writing each batch to disk before export. You turn it on with the file_storage extension. After a restart, the Gateway resends from disk to the Backend. It still has limits: a failed disk, a full disk, or a retry timeout.
Step 3: Kafka between tiers
Kafka is a durable message queue. Here the Agent produces into a Kafka topic, and the Gateway consumes from it. If the Gateway goes down, messages pile up in the topic, within Kafka's retention limits, and the Agent isn't affected. But if Kafka itself is down, the Agent needs its own queue.
Which buffer covers which failure
Each of these mechanisms is just a buffer with limits. The sending queue holds data during a backend outage, the WAL holds it through a crash, and Kafka holds it while a tier is down. Telemetry survives a failure only while some buffer still holds it.
- Backend outage: sending queue plus retries
- Collector crash: WAL on disk
- Tier down: Kafka holds messages
How data actually gets lost
The seven loss cases fall into three groups. First, overflow: the queue fills, or retries run past the time limit. Second, failure: a crash without a WAL, a full disk, or a Kafka outage. Third, a broken path: misconfiguration, or queues disabled.
- Buffer overflows or retries time out
- Buffer itself fails: memory, disk, Kafka
- Path broken: misconfig or resilience disabled
Protect the network hops
Put your buffers where the network can fail. That's Agent to Gateway, where you can use a queue, a write-ahead log, or Kafka, and Gateway to Backend, where a queue plus a WAL fits. The App to Agent link is local, so keep it light. Queues there can hurt the app. Then watch your queue size metrics.
Telemetry survives a failure only while some buffer still holds it.
Telemetry survives a failure only while some buffer still holds it. So size your queues and retries for your expected downtime, and monitor queue size against capacity.