Why can Kubernetes Pods using latest run different versions?

One Deployment, one image tag, and yet two different versions are running. Here, shop-api has three Pods. Two run v1, one runs v2. Ask kubectl, and every Pod reports the same image, shop-api:latest. Nothing in the manifest explains the split, so the tag itself is the suspect.

Follow the timeline. On Monday, CI pushes an image to the registry, and latest points to aaa. Node A pulls latest and gets aaa. Then Tuesday, CI pushes a rebuild, and latest now points to bbb. When Node C pulls latest later, it gets bbb. The tag never changed, but the content behind it did.

Look at what this manifest says. The image string is acme/shop-api:latest, and it stays identical after Tuesday's rebuild. So reapplying it triggers no rollout. Kubernetes only sees a template change, and there isn't one. Latest normally defaults to Always, but this team set IfNotPresent explicitly, so nodes trust their cache.

Follow the two restarts. The crashed Pod on Node A looks at its cache, finds latest already there, and reuses aaa, so it runs v1. The evicted Pod lands on Node C, which has nothing cached. It pulls from the registry, gets bbb, and runs v2. Same manifest, different nodes, different versions.

With imagePullPolicy IfNotPresent, a Pod restarts on a node that cached the old latest image. Which version runs?

The old cached version. IfNotPresent only checks whether an image named latest exists locally. The node finds its cached copy and never asks the registry, so it starts the old build even though latest now points somewhere else.

Switching policies doesn't fix it. With IfNotPresent, each node's cache decides, so one node reuses the old build while an empty node pulls the new one. Always asks the registry on every container start, but running Pods never re-pull. Timing still mixes versions.

A tag is a sticky note that can move. A digest is a fingerprint of one exact image. The cluster never stored which build you meant.

To prove the drift, don't trust the spec. It says latest everywhere. Ask for status imageID instead, because that's the digest each Pod actually ran. Here, two Pods show sha256 aaa and one shows bbb. Two different digests under one tag means your Deployment is running mixed versions.

The fix is the digest after the at sign. The tag stays as a readable label, but the digest decides the content, because it's derived from the bytes. Every node resolves identical content, so caching becomes safe. And a new release changes this line, which triggers a real rollout.

In practice, a commit triggers CI, and CI pushes the image under a unique tag, the commit SHA. The registry returns a digest, and you record it. That digest goes into the manifest, so you deploy by digest. Rolling back is just redeploying an older digest, with no guessing about what it contains.

Tags are movable sticky notes. Digests are fingerprints of exact images. So when Pods in one Deployment disagree, check the imageID in their status, and deploy by digest, so every node resolves the same content and each release is a real change.