What happens when a Kubernetes node runs out of disk space?

The report-job Pod just got evicted from worker-3. The node reports DiskPressure True, and it's tainted NoSchedule, so nothing new lands there. Notice who made this call. It wasn't the scheduler. The kubelet on the node decided to kill that Pod to protect itself.

A node's disk fills from four places. Image layers are shared across the whole node, and they're counted on imagefs. The other three belong to individual Pods: container logs, writable layers, and emptyDir volumes. Together, those count as each Pod's ephemeral storage, and that's what kubelet measures later.

Here's worker-3, with illustrative numbers. Image layers are the biggest consumer at 38Gi, then the OS at 19Gi. Logs add 6Gi. But report-job's emptyDir scratch data takes 25Gi by itself. Add the small Pods, and only 8Gi stays free. That's 8 percent of the disk.

Kubelet keeps polling disk stats, signals like nodefs.available, imagefs.available, and inodesFree, and compares them against the evictionHard thresholds. The default for nodefs is under 10 percent free. Worker-3 has 8 percent, so kubelet sets DiskPressure, and that adds a NoSchedule taint, blocking new Pods.

Before kubelet evicts anything, it tries cheap cleanup. It deletes dead containers, then garbage-collects unused images, and rechecks. On worker-3 that frees about 1Gi, so free space reaches only 9 percent. That's still under the 10 percent threshold, so the answer to "still low?" is yes, and eviction starts.

Kubelet ranks Pods in three steps. First, who's using more than they requested. Then lower priority goes first. Last, the biggest overuse relative to the request. Requests matter most, and with no request, every single byte counts as overuse.

On worker-3, report-job uses twenty-five gigabytes against a request of zero, so it's the biggest overdrawer and goes first. nginx-cache is over its one gigabyte request, so it's next. checkout-api stays within its request, so it's last. The disk is a shared tank, and the biggest overdrawer gets evicted.

Suppose report-job adds an ephemeral-storage request of thirty gigabytes, still using twenty-five. Which Pod would kubelet evict first now?

nginx-cache. Now report-job uses less than its request, so it falls behind any Pod exceeding its request. nginx-cache uses three gigabytes against a one gigabyte request, so it ranks first. Raw size alone does not decide eviction; usage relative to the request does.

Kubelet starts by killing report-job's containers, and the emptyDir goes with them. Free space jumps back to about 34 percent. Then kubelet marks the Pod Failed with reason Evicted through the API server. The Job controller sees the failure and creates a replacement, which lands on an untainted node, not worker-3.

Start with the Evicted events. They name the resource and the culprit, here report using 25Gi against a request of zero. The stats summary gives per-Pod ephemeral usage. Then on the node, du and crictl separate emptyDir and log usage from image layers. The event text is illustrative.

Three edge cases can surprise you. Kubelet rotates stdout logs, but not log files your app writes inside the container. A separate imagefs triggers on imagefs.available. And inode exhaustion causes DiskPressure even when free space looks fine.

The request tells the scheduler and kubelet what report-job should use, so overuse gets ranked honestly. The limit and the emptyDir sizeLimit cap it at 4Gi. Crossing either evicts only this Pod, early. If it truly needs 25Gi, give it a PersistentVolumeClaim.

Per-Pod limits aren't enough, so add node guardrails. Set containerLogMaxSize and containerLogMaxFiles so logs rotate. Image garbage collection starts at 85 percent by default, ahead of eviction. Alert on disk and inodes near 80 percent, so you act first.

Node disk is a shared tank, and kubelet evicts the biggest overdrawer. On worker-3, that was report-job, with no request and twenty-five gibibytes used. So set ephemeral-storage requests and limits on every Pod, and eviction becomes predictable instead of a surprise.