Can Kubernetes survive a node vanishing without warning?

Yes, Kubernetes can survive it, but not instantly. Here, an app called shop runs as three pods, one on each node. The Service takes user requests and spreads them across all three. That spread is what keeps the app alive when one node disappears.

Now node-2 loses power. The Service doesn't know yet, so it keeps sending some requests to that pod. Those requests time out, roughly one in three. Requests going to the pods on node-1 and node-3 still succeed normally.

A dead node doesn't send an alarm. It just goes quiet. The kubelet on node-2 normally checks in with the API server on a regular schedule. Once that stops, the node controller sees no recent heartbeats, waits out a grace period of about forty seconds, then marks node-2 NotReady.

Once node-2 is marked NotReady, its shop pod is marked not ready too. That removes it from the Service, so all traffic goes to the pods on node-1 and node-3. Errors stop, but shop is now running on only two pods.

Shop has three replicas. Node-2 just died ten seconds ago. What are users most likely seeing right now?

Some requests fail or time out. Ten seconds in, Kubernetes has not yet noticed the missing heartbeats, so the Service still sends some requests to the dead pod. The other two pods keep serving normally, so the app stays partly up rather than fully down.

Recovery isn't instant because it stacks three waits. Missed heartbeats take about forty seconds to mark the node NotReady. Then a five-minute toleration passes before eviction. Finally the new pod needs scheduling, image pull, and startup. Exact timings vary by version and configuration.

The waits are deliberate, because a silent node might just be a network blip. Once eviction happens, the ReplicaSet sees it needs one more pod, the scheduler places it on node-3, and when it's ready, it joins the Service.

One replica means total downtime when its node dies. Spread replicas across nodes, add retries to mask early errors, and keep spare capacity so replacements can schedule. Shortening the waits risks evicting healthy pods.

Kubernetes waits for missing heartbeats, then reroutes traffic and replaces the lost pods. It's recovery, not immunity. So plan for a short window of errors, and several minutes of reduced capacity, before shop is back to three replicas.