Why does kubectl drain hang on a PodDisruptionBudget?

Your drain is stuck, and it won't give up. The node got cordoned fine, but one checkout Pod refuses to be evicted. Kubectl says it would violate the disruption budget, waits five seconds, and tries again. Same error, same Pod, with no default timeout to end it.

Here's the healthy path. kubectl first asks the API Server to cordon node-1, so nothing new gets scheduled there. Then it posts an Eviction for each Pod. The API Server checks every PDB that selects that Pod. If the allowance is above zero, it deletes the Pod gracefully and returns 201 Created.

Same lanes, but now the path is blocked. kubectl asks the API Server to evict the checkout Pod, and the API Server checks the PDB. The PDB answers with zero allowed, so the API Server returns 429. The Pod is never touched. kubectl waits five seconds and tries again, and that loop is the hang.

Cordon and eviction do different jobs. Cordon marks node-1 unschedulable, so no new Pods land there, but running Pods stay put. Only eviction removes one, and only if the PDB allows it. Then the ReplicaSet notices the missing Pod and recreates it on another node.

Look at ALLOWED DISRUPTIONS. It says zero, so the eviction has no budget to spend. The pod list explains why. Three replicas exist, but the one on node-3 is in CrashLoopBackOff, not Ready. Only two count as healthy, and the PDB requires two. This output is illustrative.

Allowed disruptions is healthy Pods minus required healthy Pods. Here, two Ready minus a minAvailable of two leaves zero, so the eviction gets a 429. With maxUnavailable, required is expected replicas minus that value. Drain doesn't force anything. It asks permission for each Pod, and the budget answers.

Run the formula on each trap. One healthy replica minus one required is zero. Three minus three is zero, and maxUnavailable zero lands the same place. Our checkout case has two healthy minus two required. Different specs, same zero, same stuck drain.

On paper, checkout has three replicas and minAvailable 2, so one eviction should be fine. But allowance counts only Ready Pods. With one replica crashing, two are healthy, and the allowance drops to zero. Nothing warns you. The trap stays hidden until drain starts.

A PodDisruptionBudget only guards one path: voluntary evictions through the Eviction API, where the PDB check decides if the Pod can go. A direct delete skips that check. So does a node crash. Those losses still happen, and they shrink your healthy count, which can then block later drains.

When a drain keeps retrying, start with the error. It names the Pod being evicted. Then run kubectl get pdb and find the budget that selects it. If allowed disruptions is zero, compare Ready replicas against minAvailable. For checkout, that's two Ready against two required, which tells you which trap you're in.

Three ways to get checkout's allowance back to one. Describe the crashing replica and fix the real cause. Or scale to four replicas, so a new Ready Pod lands elsewhere. Or patch minAvailable down to one. Then watch the PDB, and drain proceeds once allowance rises.

Fixing the unready replica solves the real problem, though it takes time. Scaling up works only if the cluster has room to schedule the new Pod. Loosening the PDB is quick, but you accept less redundancy. If you change it, revert it after the upgrade.

As a last resort, --disable-eviction or deleting Pods skips the PDB check entirely. That removes the protection, and managed platforms may enforce their own PDB timeouts.

Drain asks permission for each Pod, and a PDB with zero allowance never says yes. Two Ready Pods minus minAvailable of two leaves zero allowed, so the eviction returns 429 and drain keeps retrying. Before every node upgrade, check ALLOWED DISRUPTIONS first. If it's zero, fix that before you drain.