Full lesson
Explore the full explanation, examples, and visuals at your own pace.
Terminal: status vs reality
The Pod says Running, one of one ready, zero restarts. Yet every request to orders-api returns a 503, db connection refused. Kubernetes isn't lying here. It simply never asked whether the app can do its job, so the status you're trusting is answering a much smaller question.
$ kubectl get pod orders-api-7d9f
NAME READY STATUS RESTARTS
orders-api-7d9f 1/1 Running 0
$ curl http://orders-api/orders
HTTP/1.1 503 Service Unavailable
{"error":"db connection refused"}Four layers, four different questions
Each layer answers a narrower question. Phase Running means the Pod is bound and a container started. Container running means the process is alive. Ready is True because, with no probe set, Kubernetes assumes it. The last layer, app health, is the only one that matters to clients, and nothing checks it.
What the kubelet sees vs what clients see
Follow the timeline. The kubelet starts the container, and orders-api answers that the process is alive. That's all the kubelet sees. Later, a client sends GET /orders, and orders-api tries to connect to Postgres. Postgres refuses, so the client gets a 503. That whole request path never passes through the kubelet, so it can't notice.
deployment.yaml (container spec)
Kubernetes can't see inside orders-api, so you teach it. Both /ready and /healthz are endpoints your app must implement. Three failed readiness checks mark the Pod not ready. The gotcha is that liveness deliberately ignores Postgres. Restarting the container wouldn't fix a database outage, so /healthz only proves the process responds.
readinessProbe:
httpGet:
path: /ready # app checks Postgres
port: 8080
periodSeconds: 5
failureThreshold: 3
livenessProbe:
httpGet:
path: /healthz # process responds only
port: 8080
periodSeconds: 10
failureThreshold: 3Readiness fails: Running, but no traffic
Once /ready fails three times, the kubelet marks the Pod Ready false. The EndpointSlice drops it, and the Service routes only to what's listed there, so traffic stops arriving. Now kubectl shows READY 0/1, yet STATUS still says Running. The process is alive, but it's out of rotation.
Same DB outage, two probe reactions
Same outage, two reactions. If liveness fails, the kubelet restarts the container, over and over, and the restart counter climbs while Postgres stays down. A restart can't fix a missing database. If readiness fails, the container keeps running and traffic is diverted. That's why dependency checks belong in readiness.
The orders-api database goes down for ten minutes. Which probe should fail so Kubernetes reacts sensibly?
The orders-api database goes down for ten minutes. Which probe should fail so Kubernetes reacts sensibly?
- Readiness, so traffic stops until the database returns
- Liveness, so the container restarts and reconnects
- Neither, the Pod phase will change to Failed
The orders-api database goes down for ten minutes. Which probe should fail so Kubernetes reacts sensibly?
Readiness, so traffic stops until the database returns. Restarting the container cannot bring the database back, so a liveness failure would only create a restart loop. Readiness keeps the process running but pulls it out of the Service until checks pass again. And Pod phase stays Running throughout, because the process never exits.
- Readiness, so traffic stops until the database returns
- Liveness, so the container restarts and reconnects
- Neither, the Pod phase will change to Failed
Which signal catches which failure
Each failure has its own detector. A crash shows up in container state and the restart count. A hang, like a deadlock, needs liveness to trigger a restart. Unreachable Postgres needs readiness to pull traffic, and the phase stays Running throughout.
- Process crashes: container state, restart
- Process hangs: liveness probe restarts
- Dependency down: readiness removes traffic
Monitoring sees what users experience
Probes act on one Pod at a time. Monitoring watches outcomes. Here, orders-api sends 5xx and latency metrics to Prometheus, while kube-state-metrics adds Pod Ready. When error rates spike or Pods stay unready too long, Alertmanager pages on-call. That catches slow or wrong responses probes miss.
Readiness shields users from one bad Pod, not a shared outage
If Postgres is down for every replica, every Pod goes unready, and the Service has no endpoints. Readiness can't fix that. Monitoring has to tell a human.
Check every layer, not just STATUS
Running means the process is alive, not that it can serve. So check every layer. Use readiness for dependencies like the database, liveness for hangs the process can't escape, and monitoring for what users actually experience. Each one answers a different question.