Where do Kubernetes logs go, and why does kubectl logs lose them?

The app crashed at 2am, but the crash logs are gone. Look at RESTARTS: it says 1, so the container restarted overnight. Yet kubectl logs shows only the fresh startup line. That's because kubectl logs doesn't hold history itself. It reads files kept on the node, and those files tell a specific story.

It starts inside the container. The orders-api process prints to stdout and stderr. The container runtime, containerd for example, captures that stream and writes it to a file on the node, under /var/log/pods. So your logs aren't stored in the Pod itself. They're files on one specific machine.

When you run kubectl logs on orders-api-7d9f, the request goes to the API server. The API server doesn't store any logs. It forwards the request to the kubelet on that Pod's node. The kubelet reads the local log file and streams the lines back through the API server to your terminal. So you only ever see what that one node still has.

So kubectl logs is a window, not an archive. It shows recent output from files on one node, and if those files are gone, so are the logs.

Each blank maps to the path. A file inside the container bypasses stdout, so the kubelet never captures it. A restart starts a new log file, so you see the fresh container. A replaced Pod leaves its files on the old node.

On a restart, orders-api-7d9f stays on the same node, and the kubelet keeps the crashed container's log beside the new one. So one previous log is still reachable. A replacement is different. The old Pod is gone, orders-api-x2k4 starts fresh, and kubectl can't reach the old logs.

After a restart, add --previous to read the last terminated container's output, which is where the 2am crash lives. In a multi-container Pod, -c picks between orders-api and the envoy sidecar. And if the Pod was replaced, orders-api-x2k4 is a new name with fresh history.

Even with no crash, history runs out. The kubelet writes to an active log, and when it hits a size cap, that file rotates. Older files shift back, and once the max file count is exceeded, the oldest is deleted. By default that's roughly 10Mi and 5 files. A chatty app can lose hours, and managed platforms set their own limits.

The fix is to copy logs off the node before they disappear. A log agent, usually run as a DaemonSet with a tool like Fluent Bit, tails those same node log files and ships them to a central store. You then query that store from a search UI. Many platforms offer built-in logging instead, though details vary.

Remember, kubectl logs is a window into one node's recent container files. Write to stdout and stderr, use --previous after a crash, and -c for sidecars. For anything you can't afford to lose, ship logs centrally.