Debug Kubernetes Pods with kubectl exec and kubectl debug

kubectl exec is the fast path when the container is running and the image already has the tools you need. When the image is minimal or the process exits before you can attach, kubectl debug adds an ephemeral toolbox container or clones the Pod with a safer command. Ephemeral debug images often include the curl command for quick HTTP checks from inside the Pod network namespace.

Use the Pod troubleshooting workflow when you still need phase, events, and logs before opening an interactive session.

When to Use exec or debug

Prepare the Lab Namespace

Create a running nginx Pod and a pause-style Pod that has no shell:

kubectl apply -f - <<'YAML'
apiVersion: v1
kind: Namespace
metadata:
  name: debug-lab
---
apiVersion: v1
kind: Pod
metadata:
  name: nginx
  namespace: debug-lab
spec:
  containers:
    - name: nginx
      image: nginx:1.27.0
---
apiVersion: v1
kind: Pod
metadata:
  name: pausepod
  namespace: debug-lab
spec:
  containers:
    - name: pausepod
      image: registry.k8s.io/pause:3.9
---
apiVersion: v1
kind: Pod
metadata:
  name: mc
  namespace: debug-lab
spec:
  containers:
    - name: nginx
      image: nginx:1.27.0
    - name: side
      image: busybox:1.36
      command: ["sh", "-c", "while true; do sleep 30; done"]
YAML

Wait until nginx, pausepod, and mc report Ready:

kubectl wait --for=condition=Ready pod/nginx pod/pausepod pod/mc -n debug-lab --timeout=120s
kubectl get pods -n debug-lab

Run a Command with kubectl exec

Run a single command without opening a shell:

kubectl exec nginx -n debug-lab -- ls /etc/nginx

Sample output:

conf.d
fastcgi_params
mime.types
modules
nginx.conf

Use -- before the container command so kubectl does not confuse flags:

kubectl exec nginx -n debug-lab -- printenv HOSTNAME

Sample output:

nginx

Check that the web server answers locally inside the Pod network namespace:

kubectl exec nginx -n debug-lab -- curl -s -o /dev/null -w "%{http_code}" http://127.0.0.1:80

Sample output:

200

These checks confirm the process and port work from inside the Pod before you blame Service routing.

Open an Interactive Shell

Attach an interactive terminal when you need to explore manually:

kubectl exec -it nginx -n debug-lab -- bash -c "echo shell-ok"

Sample output:

shell-ok

For a full shell session, omit the inline command:

kubectl exec -it nginx -n debug-lab -- bash

Type exit or press Ctrl+D to leave the session. The nginx container keeps running.

If bash is missing, try sh. An error such as executable file not found in $PATH means the image does not ship that shell.

Select a Container in a Multi-Container Pod

List container names first:

kubectl get pod mc -n debug-lab -o jsonpath='{.spec.containers[*].name}{"\n"}'

Sample output:

nginx side

Target the sidecar:

kubectl exec mc -n debug-lab -c side -- hostname

Sample output:

mc

Without -c, kubectl may default to the first container or print a Defaulted container notice. Sidecar layouts are easier to reason about with explicit names—see Kubernetes sidecar example.

Understand the Limits of kubectl exec

exec only works against a running container runtime sandbox. The selected container must currently be running. During CrashLoopBackOff, it may run briefly between backoff periods, but attaching with exec is timing-dependent and often unreliable.

exec is also a poor fit when:

• The container is in Error or a completed Job Pod
• The process exits before you attach
• The image has no shell (pause, distroless, or statically linked binaries only)
• Tools such as curl, ps, or dig are not in the image
• You should not install packages into the application container filesystem

When exec is not viable, use logs and events first (kubectl logs, Events and describe), then kubectl debug.

Add an Ephemeral Debug Container

Add a short-lived toolbox container to an existing Pod:

kubectl debug pausepod -n debug-lab --image=busybox:1.36 --target=pausepod --container=debugger -- sleep 3600

Sample output:

Targeting container "pausepod". If you don't see processes from this container it may be because the container runtime doesn't support this feature.

Wait until the ephemeral container is running:

kubectl wait pod/pausepod \
  -n debug-lab \
  --for='jsonpath={.status.ephemeralContainerStatuses[?(@.name=="debugger")].state.running.startedAt}' \
  --timeout=60s

Confirm the ephemeral container appears on the Pod spec:

kubectl describe pod pausepod -n debug-lab | grep -A12 'Ephemeral Containers:'

Sample output:

Ephemeral Containers:
  debugger:
    Container ID:  containerd://abc123...
    Image:         busybox:1.36
    Image ID:      docker.io/library/busybox@sha256:...
    Port:          <none>
    Host Port:     <none>
    Command:
      sleep
      3600
    State:          Running
      Started:      Sun, 26 Jul 2026 22:00:00 +0530

Key points:

• The ephemeral container joins the existing Pod and shares its network namespace.
• --target=pausepod sets the process namespace target for tools that support it.
• Ephemeral containers are for troubleshooting, not for shipping application code.

Attach interactively through the existing debugger container:

kubectl exec -it pausepod -n debug-lab -c debugger -- sh

Debug a Distroless or Minimal Image

Minimal images often run without sh. Attempting exec shows the failure clearly:

kubectl exec pausepod -n debug-lab -- sh -c "echo hi"

Sample output:

error: Internal error occurred: error executing command in container: failed to exec in container: failed to start exec "...": OCI runtime exec failed: exec failed: unable to start container process: exec: "sh": executable file not found in $PATH

Do not install packages into pausepod. Add the ephemeral debugger from the previous section, then run tests from that container:

• wget against http://127.0.0.1:<port>

A separate toolbox image containing curl would be required for curl-specific commands.

• nslookup or cat /etc/resolv.conf for DNS
• ls /proc/1/root when process-namespace sharing works for the target

The pause container keeps running unchanged while you inspect from debugger.

Create a Copy of a Pod for Debugging

Add crash Pod to lab YAML or create it before the copy step:

kubectl run crash -n debug-lab --image=busybox:1.36 --restart=Never -- sh -c 'exit 1'
kubectl wait pod/crash -n debug-lab --for=jsonpath='{.status.phase}'=Failed --timeout=60s

Because the existing crash container already uses BusyBox, --image is unnecessary when changing that container's command. Create a debug copy with --container plus a replacement command:

kubectl debug crash -n debug-lab --copy-to=crash-debug --container=crash -- sh -c 'sleep 600'
kubectl wait --for=condition=Ready pod/crash-debug -n debug-lab --timeout=60s

The original crash Pod stays failed while the copy runs:

kubectl get pod crash crash-debug -n debug-lab

Sample output:

NAME          READY   STATUS    RESTARTS   AGE
crash         0/1     Error     0          62s
crash-debug   1/1     Running   0          5s

--copy-to is ideal when you need a long-lived shell but cannot change the production Pod.

Change Images in the Copied Pod

Swap the application image on the copy without touching the original:

kubectl debug nginx -n debug-lab --copy-to=nginx-debug --set-image=nginx=busybox:1.36 --container=nginx -- sh -c "sleep 600"

Verify the copied Pod uses the debug image:

kubectl get pod nginx-debug -n debug-lab -o jsonpath='{.spec.containers[0].image}{"\n"}'

Sample output:

busybox:1.36

This creates a new Pod object. It does not update the Deployment, ReplicaSet, or Pod template on the controller.

Choose a Debug Profile

kubectl debug applies a built-in security profile to ephemeral and copied containers. The default is general:

kubectl debug nginx -n debug-lab --profile=general --image=busybox:1.36 -- sleep 3600

Profiles adjust the debug container security context and capabilities. Available profiles include general, baseline, restricted, netadmin, and sysadmin. Use general for this lab. Privileged node debugging profiles are out of scope for this Pod-focused chapter.

Common Problems

What's Next

• Build Immutable Containers in Kubernetes
• Kubernetes Audit Logging with Policy Examples
• Kubernetes Runtime Security with Falco

References

• kubectl debug — upstream command reference
• Ephemeral containers — when and how they attach to Pods
• Debug running Pods — official interactive debugging walkthrough

Summary

You practiced the two interactive paths Kubernetes offers for Pod debugging. kubectl exec remains the right choice when the container is running and the image already includes a shell and tools such as curl. Multi-container Pods need -c, and pause-style or distroless images usually force you past exec entirely.

kubectl debug covers the gaps: ephemeral containers add a toolbox to a running Pod without rebuilding the application image, while --copy-to and --set-image clone a Pod so you can keep a crashing workload alive with sleep or swap in busybox for inspection. Neither path updates the controller template—copies and ephemeral containers are deliberately isolated from production rollout semantics.

When exec fails with executable file not found, treat that as a signal to debug externally rather than installing packages into the app container. Pair interactive sessions with the inspection order in the Pod troubleshooting workflow so you attach only after events and logs point at the right container.