Why does an idle Kubernetes cluster leave Pods Pending?

Your new web Pod is stuck in Pending, yet the dashboard shows only twenty percent CPU in use. Look at the event from describe: FailedScheduling, 0 of 4 nodes available, 4 Insufficient cpu. The scheduler says there's no room, even though the cluster looks idle. The cause is hiding in what it counts.

Look at node-1 two ways. Actual usage is about 1.6 of 8 cores, mostly empty. But each of the three api Pods requests 2 cores while using only 0.2. With the system block's 1.2, requests reach 7.2 cores, and the scheduler counts that reservation, not usage.

Zoom out to all four nodes. Allocatable is 32 cores, and Used is only 6.4, about twenty percent. That's what your dashboard shows. But Requested is 28.8, so just 3.2 cores remain for scheduling, and they're split across nodes, not one big pool.

The web Pod arrives asking for 1 CPU. The scheduler checks that request against each node's unreserved room, and every node, node-1 through node-4, has only 0.8 left. None fit, so the Pod goes Pending. It never looks at live usage. The scheduler books seats by reservation, not by who's sitting there.

Each node's schedulable CPU is allocatable minus reserved requests, even if idle. And free cores on different nodes can't be combined for one Pod.

A node shows fifteen percent CPU usage, but ninety-five percent of its eight cores are requested. Will a Pod requesting one core fit?

No, only about 0.4 cores remain unreserved. The scheduler subtracts requests, not usage, from allocatable CPU. Five percent of eight cores is 0.4 cores, which is less than the one core needed. Low usage does not help, and a CPU limit does not change how the Pod is placed.

Requests can be wrong in both directions. Oversize them, like our api Pods, and capacity gets stranded, so you end up paying for idle nodes while new Pods wait. Undersize them, and nodes get overpacked. Pods contend for CPU, and under memory pressure, eviction risk goes up.

Describe node shows 7200 millicores requested, 90 percent of node-1. But kubectl top shows only 1600 millicores in use, 20 percent. That command needs metrics-server, and this output is illustrative and trimmed. Each api Pod uses about 200 millicores against a 2-core request. The scheduler only sees the first number.

Here we change the api request from two cores to 500 millicores. We base it on observed peaks, around 350 millicores, plus headroom, not the 200 millicore average. Illustratively, three api Pods then reserve 1.5 cores instead of 6, freeing 4.5 cores per node for web.

Right-sizing takes patience. Look at weeks of data, including peak traffic, then lower requests in small steps while watching latency and CPU throttling. A bursty, latency-sensitive service needs more headroom than a batch job, so there's no universal value.

The scheduler reads reservations, while dashboards show who showed up. Every node has both views, and they can disagree. Before you add nodes, compare the requests in kubectl describe node with the live usage from kubectl top. The gap usually tells you what to fix.