Full lesson
Explore the full explanation, examples, and visuals at your own pace.
Same image, two machines
The same image pulls on your laptop, but the cluster can't pull it. Up top, docker pull finishes with Pull complete. Below, the Pod api-7d9f in prod shows ErrImagePull, then settles into ImagePullBackOff. Nothing about the image changed, so the difference has to be who's doing the pulling.
$ docker pull registry.acme.io/shop/api:1.4
1.4: Pull complete
$ kubectl get pods -n prod
NAME READY STATUS RESTARTS
api-7d9f 0/1 ErrImagePull 0
api-7d9f 0/1 ImagePullBackOff 0The node pulls, not the laptop
On your laptop, the pull works because you're logged in. Then kubectl apply only sends a manifest to the API Server. It never carries your login. The API Server schedules the Pod, and the node's runtime makes its own pull from the registry, with no credentials. The registry answers 401 Unauthorized. Your laptop's login stays local, so the node was never authenticated.
Three common pull failures
Most pull failures come down to three things your laptop quietly had. A login the node never received. The exact image reference that was actually pushed. And a network path to the registry. We'll take them one at a time, starting with credentials.
- Missing or unattached pull credentials
- Wrong image name or tag
- Node cannot reach the registry
Cause 1: credentials must reach the Pod
Your login lives on your laptop, so the Pod needs its own path to credentials. The secret has to exist in prod, not default. Then the Pod references it, or the ServiceAccount carries it. That's how the node runtime gets pull credentials. Miss any link, and the runtime pulls anonymously.
Fix: create the secret in prod, reference it
First, create the secret with the namespace flag set to prod, so it lives beside the Pod. Then the Pod spec lists acme-regcred under imagePullSecrets. You could attach it to the service account instead. And use a registry token for the password, not your personal one.
kubectl create secret docker-registry acme-regcred \
--docker-server=registry.acme.io \
--docker-username=dev \
--docker-password=$TOKEN -n prod
# Pod spec
spec:
imagePullSecrets:
- name: acme-regcred
containers:
- name: api
image: registry.acme.io/shop/api:1.4Cause 2: the reference must match exactly
Even with perfect credentials, the registry only serves what was actually pushed. The manifest asks for shop/api:1.4.0, but the shelf holds 1.3 and 1.4. Close doesn't count. A locally built image may also live only in your laptop's cache, never pushed. The event will say not found.
Cause 3: nodes need a network path
Even with the right credentials and the right tag, the pull can still fail. Your laptop reaches the registry over the office network. But Node 1 and Node 2 sit in a private subnet, and their HTTPS pull goes through the firewall, where it's blocked. Missing egress, a proxy, or DNS looks the same, and the events show timeouts or no such host.
Read the events as clues
In kubectl describe, the Failed line is the clue. Here it says 401 Unauthorized, so think credentials. Not found usually points to a wrong name or tag. A timeout or no such host points to the network. One caveat: some private registries answer not found when authentication fails.
$ kubectl describe pod api-7d9f -n prod
Events:
Normal Pulling Pulling image "registry.acme.io/shop/api:1.4"
Warning Failed Failed to pull image ... 401 Unauthorized
Warning Failed Error: ErrImagePull
Normal BackOff Back-off pulling image "registry.acme.io/shop/api:1.4"
Warning Failed Error: ImagePullBackOffBackOff means waiting longer, not giving up
ImagePullBackOff doesn't mean Kubernetes gave up. The kubelet keeps retrying, but the wait grows each time. In this illustrative chart, it goes from ten seconds to twenty, forty, and on up to about three hundred, roughly a five minute cap. Once you fix the secret, the next retry succeeds, or you can recreate the Pod.
Chart values
| Retry | Seconds to wait |
|---|---|
| 1 | 10 |
| 2 | 20 |
| 3 | 40 |
| 4 | 80 |
| 5 | 160 |
| 6 | 300 |
Twist: pull succeeds, start fails
Not every image failure happens at the pull. If the image was built for arm64 on a laptop, an amd64 node can download it completely. Then the container crashes at start with exec format error. That's a startup symptom, not a pull error. The fix is a multi-platform build, or an amd64 build.
Your laptop's success proves only your laptop can pull
Each node pulls the image itself, without your laptop's login. Your laptop talked to the registry with credentials. The node runtime reached out on its own, with none, and got a 401 Unauthorized. So when a Pod fails, run kubectl describe and read the events. Then check four things: credentials, image reference, network path, and CPU architecture.


