Full lesson
Explore the full explanation, examples, and visuals at your own pace.
checkout: 10 replicas to 3
Seven Pods are about to disappear, and your users shouldn't notice. When you run kubectl scale with replicas equal to three, the Deployment updates the ReplicaSet's desired count. The ReplicaSet sees ten running against three wanted, and it has to choose seven to delete.
How the ReplicaSet picks victims
The ReplicaSet ranks Pods by how cheap they are to lose. Unhealthy ones go first, then lower deletion cost, which lets you steer. Ties fall to crowded nodes and newer Pods.
- Pending or not-ready Pods go first
- Lower pod-deletion-cost goes first
- Crowded nodes and newer Pods next
Scale and steer which Pods go
The scale command tells the Deployment to go from ten replicas to three, and the ReplicaSet picks seven victims. The annotation nudges that choice: a higher pod-deletion-cost keeps a Pod longer. It's only a best-effort hint, not a guarantee.
kubectl scale deployment checkout --replicas=3
# optional: make this Pod less likely to be removed
kubectl annotate pod checkout-7d9f-abc12 \
controller.kubernetes.io/pod-deletion-cost=100One deletion, two parallel paths
Take one of the seven Pods. Once it's marked Terminating, two things start. The EndpointSlice drops it, and kube-proxy updates its rules. Separately, the kubelet runs preStop, then sends SIGTERM. Nothing orders these two paths. They run side by side, and neither waits for the other.
The race users can feel
If SIGTERM arrives before routing updates propagate, new requests still reach a Pod that's closing. The preStop sleep delays shutdown, so routing wins the race. The mental model is simple: unlist the Pod first, then drain it.
The checkout app exits instantly on SIGTERM and has no preStop hook. What might users see during scale-down?
The checkout app exits instantly on SIGTERM and has no preStop hook. What might users see during scale-down?
- A few failed requests right after scale-down
- Nothing, endpoints are always removed first
- The three remaining Pods restart
The checkout app exits instantly on SIGTERM and has no preStop hook. What might users see during scale-down?
A few failed requests right after scale-down. Endpoint removal and SIGTERM start in parallel, so kube-proxy may still send new requests to a Pod that already exited. Kubernetes does not guarantee routing updates finish first, which is why a short preStop delay helps.
- A few failed requests right after scale-down
- Nothing, endpoints are always removed first
- The three remaining Pods restart
checkout Pod spec
The preStop hook sleeps five seconds, giving kube-proxy time to stop routing new requests. Then SIGTERM arrives. That thirty second grace period covers the sleep plus app shutdown together. One gotcha: exec needs a sleep binary, so minimal images may not have one.
spec:
terminationGracePeriodSeconds: 30
containers:
- name: checkout
lifecycle:
preStop:
exec:
command: ["sleep", "5"]Inside the 30 second grace period
After SIGTERM, the app stops accepting new work and finishes in-flight checkouts, then exits cleanly. If draining runs past thirty seconds, SIGKILL drops whatever remains. Keep-alive connections won't end by themselves, so the app should close them too.
Seven Pods gone, zero dropped requests
Scale-down is a race. Unlist the Pod from the Service, then let it drain. So add a preStop delay, handle SIGTERM in your app, and all seven Pods leave without a single dropped request.