Full lesson
Explore the full explanation, examples, and visuals at your own pace.
Part 1
After scaling to three replicas, users start getting randomly logged out. Here's one user. They post to login, the Service forwards it to Pod A, and Pod A keeps the session in its own memory, then sets cookie sid equals abc. The next click, to cart, gets forwarded to Pod B. Pod B has never seen that sid, so it redirects to login.
The Service balances traffic, blind to sessions
The cause is the Service. It receives the user's request with sid=abc and spreads traffic across Pod A, Pod B, and Pod C. It doesn't read cookies, and it has no idea where a session lives. With one Pod, that never mattered. With three, most requests land somewhere other than Pod A.
Rolling update: every deploy logs everyone out
Now the deploy. The Deployment starts a new Pod, Pod D, running the new version. Then it terminates Pod A, and everything in Pod A's memory is gone, including session abc. The user's next request, GET /cart with sid=abc, hits Pod D, which has never seen that session, so it redirects to login. The same happens as B and C get replaced, so every deploy logs everyone out.
Pods are replaceable: their memory and local disk are temporary.
Kubernetes freely kills, reschedules, and replaces Pods. Their memory and local disk are temporary, so nothing stored only inside one Pod should be trusted to last.
Same bug, other per-Pod state
Sessions aren't the only victim. Any per-Pod state breaks the same way. Caches drift out of sync between Pods. A limit of 100 requests quietly becomes about 300 across three Pods. And uploaded files saved locally vanish, or return a 404 from other Pods.
- In-memory caches diverge between Pods
- Rate limit counters split three ways
- Uploaded files saved on local disk
Fix 1: shared session store
The first fix is to move sessions out of the Pods and into a shared store, here Redis. Now any Pod can read or write session abc. So when /cart lands on Pod B, it finds the session and the user stays logged in. Redeploys stop wiping sessions. The cost is an extra network hop, and Redis must stay highly available.
Fix 2: signed tokens any Pod can verify
Here's the same login, but Pod A hands back a signed token instead of storing a session. The user sends it with GET /cart, the Service picks Pod B, and Pod B just verifies the signature using the shared signing key. No lookup needed. The trade-offs: revoking a token early is hard, and every Pod needs that key, so you have to distribute and rotate it carefully.
Fix 3: sessionAffinity ClientIP and its pitfalls
Setting sessionAffinity to ClientIP tells the Service to pin each source IP to one Pod. But look at who shares an IP. Many users behind an office NAT look like one client, so they all land on Pod A. Behind an Ingress, the Service may only see the Ingress's IP, so everyone gets pinned together.
More affinity limits
Cookie-based stickiness isn't a Service feature. It depends on your Ingress controller and its annotations. And if a Pod restarts or gets replaced, its sessions are still gone. Load can also skew unevenly across Pods. Affinity is an optimization, not a fix.
- Cookie affinity depends on Ingress controller
- Pod restart still loses its sessions
- Uneven load across Pods
Any Pod must serve any request; Pod memory is disposable.
Any Pod must serve any request; Pod memory is disposable. So keep sessions in a shared store or signed tokens, and treat affinity as an optimization.