Full lesson
Explore the full explanation, examples, and visuals at your own pace.
Load test running, CPU climbing
The load test is running, CPU is climbing, and the replicas stay at 2. Look at the TARGETS column. Instead of a percentage, it says unknown over 70 percent. The exact output format can vary by version, but that unknown means the HPA can't compute a metric, so it never scales.
$ kubectl get hpa web
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS
web Deployment/web <unknown>/70% 2 5 2The HPA control loop
The HPA is a pipeline. metrics-server reports Pod CPU usage in millicores. The HPA divides that by the request to get utilization, compares it with 70%, and computes desired replicas. If an earlier stage fails, nothing reaches the end, so the replica count never changes.
<unknown> = cannot calculate the metric, not low load
Unknown doesn't mean low load. It means the HPA has no number to compare against 70 percent, so it can't decide anything, and replicas stay put.
Break 1: no metrics source
The first break happens before any math. The HPA controller asks the Metrics API for Pod CPU, and that request gets forwarded to metrics-server. But metrics-server is missing or failing, so the Metrics API has nothing to return. No usage data means no utilization, no decision, and replicas stay unchanged at the minimum of two.
Break 2: a container without a CPU request
Now suppose metrics-server is healthy, but the log-agent sidecar has no CPU request. Web contributes 400m, while log-agent contributes nothing. The HPA computes utilization against the Pod's total request, and that sum is incomplete. With no valid denominator, the percentage is undefined, even though web has a request.
deployment.yaml: request CPU on every container
The fix is small. Keep the 400m request on web, and add a 100m request to log-agent. Now the Pod request totals 500m, so the HPA has a complete denominator. Watch for injected sidecars, and for LimitRange defaults that can add or hide this field.
containers:
- name: web
image: web:1.4
resources:
requests:
cpu: 400m
- name: log-agent
image: log-agent:2.0
resources:
requests:
cpu: 100m # was missing: HPA showed <unknown>Healthy pipeline, same load test
Same load test, now with the request in place. Usage is 700m, and the Pod requests 500m, so utilization is 140%. The HPA compares that to the 70% target. Two replicas times 140 over 70 gives 4. TARGETS now shows 140% over 70%, and scaling begins.
Same 700m usage, three different denominators
The same 700m of usage gives three different answers, depending on the denominator. Against the 500m request, it's 140 percent. Against the 1000m limit, 70. Against the 4000m node, 17.5. The HPA uses requests, so request size decides when scaling starts.
Chart values
| Denominator | Percent |
|---|---|
| vs request 500m | 140 |
| vs limit 1000m | 70 |
| vs node 4000m | 17.5 |
Working HPA, but capped: TARGETS 112%/70%
Here the metric works. Utilization is 112% against a 70% target, so the math says ceil of 5 times 112 over 70, which is 8 replicas. But maxReplicas is 5, so Running stays at 5. A real number above target at max isn't a bug. It's a ceiling.
Chart values
| Label | Replicas |
|---|---|
| Calculated | 8 |
| maxReplicas | 5 |
| Running | 5 |
Read the TARGETS column
Three states, three meanings. Unknown means the pipeline is broken, so no decision is even made. A real percentage with replicas rising is healthy scaling. A real percentage above target at max replicas means the HPA is capped, and ScalingLimited explains it.
- <unknown>/70%: metric cannot be computed
- 140%/70%, replicas rising: healthy scaling
- 112%/70% at max: capped, ScalingLimited
kubectl describe hpa web (illustrative, wording varies)
In the conditions, ScalingActive is False with reason FailedGetResourceMetric. That means the HPA has no metric to work with. Here ScalingLimited is False, so no min or max cap is involved. If it were True, a cap would be at play. The event names the cause, though exact wording varies by version.
Conditions:
Type Status Reason
AbleToScale True SucceededGetScale
ScalingActive False FailedGetResourceMetric
ScalingLimited False DesiredWithinRange
Events:
Warning FailedGetResourceMetric
failed to get cpu utilization:
missing request for cpuDiagnostic path
Start with describe hpa and read the conditions and events. Then run kubectl top pods. If data comes back, metrics-server works. Next, check requests on every container, including the sidecar. Once both exist, the metric computes. Custom and external metrics skip this path, because they depend on their own adapters.
Every stage must produce a value
HPA math is usage over requests. If either one is missing, the answer is unknown. Every stage in this pipeline, from metrics-server to desired replicas, has to produce a value. So when scaling stalls, check the metrics and every container's request first, before you start tuning the 70 percent target.