Full lesson
Explore the full explanation, examples, and visuals at your own pace.
Part 1
When a Pod dies mid-request, that request simply fails. Here, the client sends POST /orders to the Service, which routes the connection to pod-b. Then pod-b crashes. The client gets a reset, a timeout, or a 502 from a proxy. Kubernetes doesn't replay the request for you.
Before the crash: one Service, three Pods
Here's the setup before anything breaks. The client sends POST /orders to the Service named orders. That Service is a stable address, backed by a list of ready Pods: pod-a, pod-b, and pod-c. It spreads connections across them, and this one happened to land on pod-b.
The target is chosen once, when the connection opens
When the client opens a connection, the kube-proxy rules pick one Pod, and here that's pod-b. They pick once. After that, data flows straight between the client and pod-b. So when pod-b crashes, the connection is lost, and the request has nowhere else to go.
Kubernetes reroutes new connections, never ones already in flight.
Rerouting only helps new connections. The half-processed order on pod-b is gone, and unless the client retries, nobody will ever finish it.
Removing the dead Pod from rotation
The kubelet notices pod-b crashed and reports it not ready to the API server. The API server removes pod-b from the EndpointSlice, and kube-proxy updates its routing rules. This takes a moment, so traffic can still reach pod-b briefly. If the whole node dies, the delay is longer.
After pod-b is removed from the endpoint list, the client retries POST /orders. Where does the retry go?
After pod-b is removed from the endpoint list, the client retries POST /orders. Where does the retry go?
- To pod-a or pod-c
- Back to pod-b once it restarts
- Into a queue until pod-b recovers
After pod-b is removed from the endpoint list, the client retries POST /orders. Where does the retry go?
To pod-a or pod-c. The retry opens a brand new connection, and the Service only routes new connections to Pods still in its ready endpoint list. Kubernetes does not hold or queue requests for a crashed Pod, so pod-a or pod-c handles it.
- To pod-a or pod-c
- Back to pod-b once it restarts
- Into a queue until pod-b recovers
New traffic reaches only healthy Pods
Here's the result. The client's retry hits the Service, which sends the new connection to pod-a or pod-c, since only ready Pods are listed. Meanwhile the ReplicaSet creates pod-d. It doesn't get traffic right away. The Service adds it only after its readiness probe passes.
Protecting in-flight requests
Retries recover the lost order, but only safely if the request is idempotent, otherwise you risk duplicate orders. Readiness probes protect new traffic. Graceful shutdown helps planned stops, but it can't save a request from a sudden crash.
- Client retries, safest for idempotent requests
- Readiness probes gate new traffic
- Graceful shutdown drains planned terminations
Lost connection, rerouted retry
Dead Pods drop open connections, and Services steer only new ones elsewhere. Kubernetes won't replay the lost POST to orders. So design your clients to retry safely, with idempotent operations, and the next connection will find a healthy Pod.