Full lesson
Explore the full explanation, examples, and visuals at your own pace.
Part 1
You ran kubectl rollout undo, and the errors didn't drop. The chart shows the error rate jumping to 14 percent when v1.5 ships, then staying near 13 after the undo. The Pods restarted with the old image, so the image isn't the problem. Something else didn't roll back.
Chart values
| Timeline | Error % |
|---|---|
| v1.4 live | 0.4 |
| v1.5 ships | 14 |
| after undo | 13 |
| 10 min later | 13.5 |
Release v1.5: two changes
Release v1.5 changed two things. In the Deployment, the image moved from 1.4 to 1.5. Separately, the ConfigMap's DB_HOST now points at db-new.internal, which is wrong. Notice the Deployment never contains that value. It only references checkout-config by name, while the bad host lives inside the ConfigMap.
# deployment.yaml (excerpt)
containers:
- name: checkout
image: shop/checkout:1.5 # was 1.4
envFrom:
- configMapRef:
name: checkout-config
---
# configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: checkout-config
data:
DB_HOST: db-new.internal # was db-primary.internalTemplate snapshots vs one live ConfigMap
The Deployment keeps its history as ReplicaSets, rev1 and rev2. Each one snapshots a Pod template: the image, env literals, and the ConfigMap's name. But checkout-config isn't part of either snapshot. Both point to it by name, so there's one live copy, holding only its latest data.
What undo actually does
Follow the undo. kubectl tells the Deployment to go back to revision 1. The Deployment scales up RS rev1, and that template becomes a new revision. The New Pod starts with image 1.4, exactly as before. Then it reads checkout-config by name, and gets DB_HOST equal to db-new. That's the latest data, not what revision 1 ever saw.
Rollout history versions the Pod template, not the objects it references.
Rollout history versions the Pod template, nothing else. Undo rewinds the template, but checkout-config is referenced by name, so it still holds today's data.
Three distinctions to remember
A name reference resolves to whatever the ConfigMap holds right now, and env vars are read when a Pod starts. Secrets work the same way, so they carry the same risk. And a ConfigMap-only edit creates no revision, so history has nothing to undo.
- A name reference resolves to current contents
- Secrets behave exactly the same way
- ConfigMap-only edits create no new revision
Fix 1: versioned ConfigMap names
With Kustomize, the configMapGenerator doesn't keep the name checkout-config. It appends a hash of the contents and rewrites the Deployment's reference to match. So changing DB_HOST changes the name, which changes the Pod template. That creates a real revision, and rollout undo has something meaningful to restore.
# kustomization.yaml
resources:
- deployment.yaml
configMapGenerator:
- name: checkout-config
literals:
- DB_HOST=db-primary.internal
# generated name: checkout-config-<hash>
# Deployment reference rewritten to matchEach revision pins its own config
Now each ReplicaSet points at its own hashed ConfigMap. Rev1 refers to config-a1b2, rev2 refers to config-c3d4. So when you undo to rev1, Pods go back to config-a1b2, with the good DB_HOST. The trade-off is that old ConfigMaps have to stay around, so pruning needs care.
Other approaches and trade-offs
You have options. Setting immutable true on a ConfigMap blocks silent edits, but every change needs a new name. Reverting and reapplying all manifests from Git restores everything together, though it's slower. Hashed names make undo safe, but old copies pile up.
- Immutable ConfigMaps: no in-place edits, new names
- Git revert: restores everything, slower path
- Hashed names: safe undo, old copies pile up
Check everything the release changed
Rollback rewinds the Pod template. Named ConfigMaps stay at today's data. So with checkout, image 1.4 comes back, but DB_HOST still points at db-new.internal. Before you run undo, check every object the release changed, not just the Deployment.