Why is my Kubernetes container OOMKilled when the node has free memory?

Your orders-api container just got OOMKilled, and the node still has about ten gigabytes free. Look at the chart. Memory climbs steadily from 200 to 512 mebibytes over an hour, then the kill lands. The node's free memory never entered the decision. The container's own limit did.

There are two separate budgets here. The node has sixteen gibibytes, and it hosts the pod. But the container runs inside a cgroup, and that cgroup enforces the 512Mi limit, capping the app process. The app can't borrow from the node's free memory. Past 512Mi, it's over its own budget.

On the left, the container limit is a fence around one app. The kernel enforces it, and only that container gets killed. On the right, node memory is a big field with lots of empty space. Node pressure is handled differently: kubelet evicts whole Pods. Two budgets, two mechanisms.

Follow orders-api as it crosses its limit. The app tries to allocate past 512Mi, and the kernel kills the process. That's exit code 137. The kubelet hears about it through the container runtime, reports OOMKilled to the API server, and, since restartPolicy is Always, starts the container again. Nothing here involved node free memory.

The scheduler only reserved the 256Mi request when placing the Pod. Plenty of node memory stays free, but the 512Mi limit still caps the container.

Watch the sawtooth. Memory climbs from 200Mi to 512Mi, the container gets killed, and the restart drops it back to 200Mi. But it's the same code, so it leaks the same five Mi per minute and hits the limit again. A restart clears the symptom, not the cause.

orders-api still leaks about five Mi per minute. If you raise its limit to one Gi, what happens?

OOMKills still happen, just later. The leak keeps growing no matter where the fence sits. With a one Gi limit it simply takes about two and a half hours instead of one to hit the wall. A bigger limit buys time, but only fixing the leak flattens the line.

Follow the cycle. The leak grows until usage hits 512Mi, the kernel kills the process, and kubelet waits before restarting it. Memory resets, and the loop begins again. Repeated failures show up as CrashLoopBackOff, with waits that grow up to about five minutes. Deleting the Pod just starts the same loop elsewhere.

Before you touch the limit, confirm the kill. Pod describe should show last state OOMKilled, exit code 137. Then chart working-set memory against the limit. A steady climb means a leak. A flat plateau above the limit means the limit is too small.

Doubling the limit to 1Gi feels like a fix, but the line keeps climbing. The kill just lands around two and a half hours instead of one. Profiling the heap and fixing the leak flattens the line. Raise limits only when the app genuinely needs more steady memory.

A container dies at its own memory limit, no matter how much the node has free. Restarts only reset the symptom. So look at the memory shape: a steady climb means a leak, and the fix belongs in the code, not the restart.