Why kubectl exec fails with 'executable not found' and how to debug

Running kubectl exec with bash into the checkout Pod fails with executable file not found, and exit code 126. The cause is simple. The app container's image, a distroless one, doesn't contain bash at all. Exec can only run what's already inside that image, so there's nothing for it to start.

Here's the path your command takes. kubectl sends the exec request to the API server, which forwards the stream to the kubelet. The kubelet then asks for bash to be spawned inside the app container. But exec brings no tools of its own. It looks for bash in the container's own image filesystem, doesn't find it in PATH, and the error returns as exit code 126.

Here's the same checkout app on two images. On distroless, the container holds just the app binary and CA certificates, so there's no bash and no sh at all. Alpine is different. It ships busybox sh, but not bash. So exec with sh succeeds on Alpine, while bash fails on both.

Distroless images leave out the shell on purpose. They're smaller, and an attacker who gets in has fewer tools to use. That's a feature, not a bug.

Missing tools aren't the only problem. Now picture checkout in CrashLoopBackOff. The app starts, crashes, and exits. Kubernetes waits, with a delay that grows, then restarts it. Because the container is mostly exited or waiting in backoff, exec has nothing running to join. It can fail even if the image has a shell.

This command adds a new busybox container to the live checkout Pod, without restarting app. The target flag points it at app, so it can join app's process namespace, and ps may show app's processes if the runtime supports that. Because the Pod shares one network, localhost reaches app's health endpoint.

exec borrows the tools already baked into the app's image. kubectl debug brings its own.

The ephemeral busybox container sits beside the app container, and both share the Pod network. With --target and runtime support, busybox can also see the app's processes. So you don't need bash inside the app image.

Ephemeral containers are one-way. Once you add busybox to the checkout Pod, you can't remove it. If it exits, Kubernetes won't restart it. It only goes away when the Pod is deleted. They also can't define ports or probes.

When checkout keeps crashing, kubectl debug with --copy-to creates a new Pod, checkout-debug, and leaves the original untouched. You can add busybox with shared processes, or, if the image has sh, override the command so app stays up. It's a separate Pod, so delete it when you're done.

These options have limits. Ephemeral containers are stable from Kubernetes 1.25, so older clusters may lack them. Sharing processes with --target depends on runtime support. And RBAC permissions on pods/exec and ephemeralcontainers can block these commands, so check access first.

Exec borrows the image's tools; kubectl debug brings its own. So when exec fails, try sh before bash. If the Pod is running, attach a busybox container with kubectl debug. If it's crashing, debug a copy with --copy-to.