Why does kubectl apply undo HPA scaling and cut capacity?

A routine kubectl apply cut our API from ten Pods to three, right in the middle of a traffic peak. On the chart, replicas climb to 10, drop to 3 at the apply, then only return to 10 minutes later. Latency jumped in that gap, before the HPA recovered capacity.

Both the HPA and kubectl apply write the same field, spec.replicas. First, the HPA sets it to 10 for the peak. Then apply sets it back to 3, because the manifest still says 3. The Deployment obeys and terminates 7 Pods, and latency jumps. The HPA writes 10 again, but those 7 new Pods must start and become Ready before they help.

Once an HPA manages a workload, the replica count is computed from metrics. It's an output. So the number in your manifest is just a stale snapshot.

Look at spec.replicas in the middle. Two writers feed it. Under high load, the HPA writes 10. The manifest, on apply, writes 3. Kubernetes has no idea which value is right. The Deployment controller just reconciles the Pods to whatever was written last, and right after the apply, that's the manifest's 3.

Look at the line. Ready Pods drop from 10 to 3 at the apply, then stay flat for a couple of minutes. The HPA has to notice the load, and new Pods must schedule, pull images, start, and pass readiness. Only then do they climb back to 10. These timings are illustrative; yours depend on your app.

Now add a GitOps tool that syncs continuously. It sets replicas to 3, then the HPA sets it back to 10. The Deployment's live state no longer matches Git, so the tool reports drift. With self-heal on, it resets replicas to 3 again, and the HPA answers with 10 again. The dip isn't a one-time cost of a deploy anymore. It can repeat.

The fix is to give the HPA sole ownership of the replica count. Drop spec.replicas from the manifest, and let minReplicas and maxReplicas set the bounds. Then a deploy changes configuration without resizing the Deployment, because only the HPA writes that number.

Removing replicas isn't a casual edit. With client-side apply, simply deleting the field can trigger a one-time scale change. So follow the documented Kubernetes migration steps for your apply mode, because server-side apply tracks field ownership differently.

If a GitOps tool owns your manifests, you can tell it to ignore spec.replicas when it diffs and syncs. Argo CD does this with ignoreDifferences. Exact behavior varies by tool, version, and apply mode, so test it on your own setup.

Replicas is one field with two writers, and only one should write it. Audit your HPA-managed workloads for a hardcoded replicas value in the manifest.