Why do Kubernetes rolling updates cause brief 502 and 503 errors?

Every time orders-api deploys, a few seconds of requests fail with 502 and 503 errors. This chart shows failed requests per second across one rollout, with peaks near twenty and forty seconds. The numbers are illustrative, but the pattern is real. A rolling update on its own doesn't prevent it.

The errors come from two timing races. One happens when old Pods stop, the other when new Pods start. Each one leaves a small window for failed requests.

When orders-api replaces an old Pod, the control plane sends SIGTERM to the Pod and tells the Ingress to remove its endpoint, at the same time. The app closes its listener immediately. Endpoint removal takes time to propagate, though. So a client's GET /orders still reaches the Ingress, which forwards it to the old Pod. Connection refused, and the client sees a 502.

Startup has its own race. The control plane starts the container on the new Pod, and with no real readiness probe, Kubernetes treats it as Ready right away. So the endpoint gets added to the Ingress while the app is still warming up. A client sends GET /orders, the Ingress forwards it, the Pod isn't ready, and the client gets a 503.

Kubernetes swaps Pods, but routing updates lag behind. Errors appear in that gap, and with four replacements, orders-api opens both windows four times every deploy.

A new orders-api Pod has no readiness probe, and its app needs fifteen seconds to warm up. What most likely happens?

It gets traffic early and some requests fail. Without a readiness probe, Kubernetes treats the Pod as Ready once its container starts. The endpoint is added right away, so requests arrive during warmup and fail. Kubernetes does not check whether the app can actually serve unless a probe tells it.

The fix for startup is a readiness probe that tells the truth. It should pass only when orders-api can actually serve, say once its dependencies are loaded. Then the endpoint is added after warmup, not at container start. Keep in mind the probe runs on a period, so Ready arrives slightly later than the app is able.

For shutdown, we buy time before the app reacts. When the Pod is deleted, a short preStop delay lets endpoint removal propagate, so the Ingress stops sending GET /orders its way. Then SIGTERM arrives, the app stops new connections and drains in-flight requests. The grace period has to cover the delay and the drain.

Retries are the last layer. If a GET /orders hits a closing Pod, a retry at the client or proxy can land on a healthy replica. But retry only idempotent requests, never blind POSTs, and cap attempts, because retries add load.

How long the gap lasts depends on your stack: how quickly the Ingress controller syncs endpoints, how often the cloud load balancer health checks run, and how fast kube-proxy or the CNI updates routing. So measure before tuning delays. These fixes reduce risk, not guarantee zero downtime.

Deploy errors happen when routing state and Pod state change at different speeds. Readiness probes narrow the startup race. The preStop delay and graceful drain narrow the shutdown race. Retries absorb the rest. Change one thing, then measure the error spike after each deploy.