Kubernetes Secret updates: why do running pods keep old passwords?

Updating a Secret doesn't restart a single pod. After rotation, the environment variable still holds v1, while the mounted file eventually shows v2. Same Secret, two different outcomes. And even the file with v2 may not save you, because the app might never look at it again.

Here's our running example. The Secret db-creds holds one key, password, currently s3cret-v1. The orders-api container consumes it twice: as the env var DB_PASSWORD through secretKeyRef, and as a file under /etc/db. Notice there's no subPath on that mount. That detail matters later.

On rotation day, the operator sets the Postgres password to v2, then updates db-creds through the API server. It replies that v2 is stored, and that's all. No pod restarts. Later, not instantly, the kubelet syncs the Secret, and that refresh only reaches volumes. The running process hasn't been told anything.

The env var path is a one-time copy. When the container starts, the kubelet resolves the secretKeyRef and puts the value into the process environment. Nobody can edit that environment from outside, so DB_PASSWORD stays v1 in orders-api until a new container starts.

Volume mounts behave differently. When db-creds changes, the kubelet picks it up on a periodic sync, then swaps the file in place atomically. So /etc/db/password becomes v2, but only after a delay. One caveat: a subPath mount never updates. The open question is when orders-api reads it.

The file under /etc/db now says v2, but orders-api read it once at startup. That value sits in memory as v1, and the connection pool was built from it. Env vars snapshot, mounted files refresh, but your app may never look again.

orders-api reads the mounted password file once at startup. The Secret is updated to v2. Ten minutes later, what do new database connections use?

The old password v1. The kubelet did refresh the file to v2, but the app copied v1 into memory at startup and never reads the file again. Volume mounts only update the delivery; the process must re-read the file or restart to use the new password.

Look at what the logs show. Reused connections keep working, because they authenticated with v1 earlier. Then the pool opens a new connection, sends v1, and Postgres rejects it. That's why the failure is delayed, sometimes by minutes or hours, and looks unrelated to the rotation.

The rollout restart covers both paths, because new containers resolve the env var again and read the fresh file. If only the volume matters, check the file with sha256sum, which proves it changed without printing the secret. Then the app must re-read it and rebuild its pool.

Order matters here. Add v2 to the database first, while v1 still works. Then update the Secret, and restart or reload orders-api. Only when every pod is on v2 do you revoke v1. A second database user is one way to keep both valid.

Env vars are a startup snapshot. Mounted files refresh, but your app may never look at them again. So when you rotate a password, overlap the old and new, switch every pod, and revoke the old one last.