An image built on distroless or scratch has no shell, no curl, no ps, by deliberate security choice: fewer embedded binaries means less attack surface if the container gets compromised. That choice’s price is paid at debugging time: kubectl exec -it my-pod -- sh fails with exec: "sh": executable file not found, exactly when a pod is misbehaving and a closer look matters most.

The ephemeral container: a debug tool attached live

An ephemeral container attaches to an already-running pod, sharing its network and process namespaces, without restarting any existing container or permanently modifying the pod’s spec.

kubectl debug -it my-pod \
  --image=busybox:1.36 \
  --target=app-container

--target specifies which of the pod’s containers to share the process namespace with: without it, the debug container sees only its own processes, not the application container’s it’s meant to inspect. Once attached, busybox’s full toolbox (or any preferred debug image) becomes available, without a single byte added to the production image.

What sets an ephemeral container apart from a plain exec

kubectl exec runs inside an already-existing container, with only the binaries it already ships. kubectl debug adds a new one to the pod, temporarily, with whichever image you choose. It’s the difference between searching for a tool that doesn’t exist in an empty room and bringing your own toolbox without having to move the room.

Copying an entire pod for more invasive debugging

Some diagnostics require changing environment variables or adding system capabilities the original pod doesn’t have: kubectl debug then offers creating a copy of the pod instead of attaching an ephemeral container.

kubectl debug my-pod -it \
  --copy-to=my-pod-debug \
  --container=app-container \
  --set-image=app-container=my-image:debug

That copy runs alongside the untouched original, which allows experimenting (swapping an image, adding a capability) without ever risking the actual production pod. Once the diagnostic is done, the copy gets deleted, leaving no trace on the original.

Debugging a node directly

The same mechanism extends to the node itself, useful when the problem isn’t in a pod but in the node hosting it:

kubectl debug node/my-node -it --image=busybox:1.36

This command creates a privileged pod on the targeted node, with its filesystem mounted under /host, which allows inspecting node state directly (system logs, network configuration) without going through SSH when SSH access isn’t configured or available.

What this changes about image strategy

Knowing kubectl debug exists changes the security calculus around minimal images: the fear of “losing the ability to debug,” which pushes some teams to keep a shell in their production images (at the cost of a larger attack surface), no longer has a technical justification once a live diagnostic tool, entirely separate from the image itself, is available.

Takeaway

A shell-less distroless image isn’t an obstacle to debugging, just the absence of a shell that was never needed in production in the first place. Ephemeral containers attach a full debug toolkit to an already-running pod, without restarting it; --copy-to allows more invasive debugging without ever risking the original; the same mechanism extends to nodes themselves. This reflex directly complements the diagnostic tools already mentioned in the CrashLoopBackOff glossary entry, for the cases where the image itself leaves no direct access at all. Shrinking image attack surface while keeping real diagnostic tooling is part of what gets decided during a CI/CD industrialization.