Kubernetes Pod Troubleshooting Workflow

You already know the Pod is unhealthy. This workflow tells you what to check next so you do not skip scheduling while chasing logs, or chase Service DNS while the container never pulled its image.

The examples use deliberately failing workloads in the pod-troubleshoot-lab namespace: a crashing Deployment and a Deployment with a bad image tag. Command details for describe, events, and logs are covered in kubectl logs, Events and describe—this page focuses on order and routing.

Quick reference

Walk this sequence on every unhealthy Pod. Replace NS and POD with your namespace and Pod name.

Prepare the Pod Troubleshooting Lab

Create two Deployments: one that exits on startup and one with a missing image tag. A Service selects the crashing Deployment so you can inspect EndpointSlice readiness later.

kubectl apply -f - <<'YAML'
apiVersion: v1
kind: Namespace
metadata:
  name: pod-troubleshoot-lab
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: crash-web
  namespace: pod-troubleshoot-lab
spec:
  replicas: 1
  selector:
    matchLabels:
      app: crash-web
  template:
    metadata:
      labels:
        app: crash-web
    spec:
      containers:
        - name: app
          image: busybox:1.36
          command: ["sh", "-c", "echo start; exit 1"]
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: bad-image
  namespace: pod-troubleshoot-lab
spec:
  replicas: 1
  selector:
    matchLabels:
      app: bad-image
  template:
    metadata:
      labels:
        app: bad-image
    spec:
      containers:
        - name: app
          image: nginx:nonexistent-tag-xyz
---
apiVersion: v1
kind: Service
metadata:
  name: crash-web
  namespace: pod-troubleshoot-lab
spec:
  selector:
    app: crash-web
  ports:
    - port: 80
      targetPort: 80
YAML

Sample output:

namespace/pod-troubleshoot-lab created
deployment.apps/crash-web created
deployment.apps/bad-image created
service/crash-web created

Wait for the Pods to exist:

kubectl wait --for=create pod -n pod-troubleshoot-lab -l app=crash-web --timeout=60s

kubectl wait --for=create pod -n pod-troubleshoot-lab -l app=bad-image --timeout=60s

Set the variables:

POD=$(kubectl get pods -n pod-troubleshoot-lab -l app=crash-web -o jsonpath='{.items[0].metadata.name}')

BAD=$(kubectl get pods -n pod-troubleshoot-lab -l app=bad-image -o jsonpath='{.items[0].metadata.name}')

Wait for the stable failure reasons:

kubectl wait pod "$POD" -n pod-troubleshoot-lab --for='jsonpath={.status.containerStatuses[0].state.waiting.reason}=CrashLoopBackOff' --timeout=120s

kubectl wait pod "$BAD" -n pod-troubleshoot-lab --for='jsonpath={.status.containerStatuses[0].state.waiting.reason}=ImagePullBackOff' --timeout=120s

CrashLoopBackOff is the kubelet's restart-backoff state after repeated container failures, while ImagePullBackOff indicates that the kubelet is backing off repeated pull attempts.

Identify and Classify the Failure

Find the Pod and controller

Capture identity and context before you open logs.

List Pods with node placement and restart count:

kubectl get pods -n pod-troubleshoot-lab -o wide

Sample output:

NAME                         READY   STATUS              RESTARTS   AGE   IP             NODE
bad-image-xxxxxxxxxx-aaaaa   0/1     ImagePullBackOff    0          35s   192.168.5.62   worker01
crash-web-xxxxxxxxxx-bbbbb   0/1     CrashLoopBackOff    3          35s   192.168.5.46   worker01

Restart counts and generated names vary with backoff timing.

Find the controller that created the Pod:

kubectl get pod "$POD" -n pod-troubleshoot-lab -o jsonpath='{.metadata.ownerReferences[0].kind}{"/"}{.metadata.ownerReferences[0].name}{"\n"}'

Sample output:

ReplicaSet/crash-web-5649dbcd64

A ReplicaSet owner means you should also inspect the parent Deployment when fixes require template changes.

Confirm the workload controller:

kubectl get deployment -n pod-troubleshoot-lab

Owner, node, and age provide context. Use STATUS, container state, restart count, and events to determine whether the failure is scheduling, image pull, startup, or probe related.

Compare phase, STATUS, readiness, and restarts

Phase is the Pod lifecycle field: Pending, Running, Succeeded, Failed, or Unknown.

STATUS in kubectl get is a convenience column. It often shows container reasons such as CrashLoopBackOff, ImagePullBackOff, ContainerCreating, ErrImagePull, or Terminating. Phase can stay Running while STATUS shows CrashLoopBackOff because the Pod is still on a node and containers keep restarting.

CrashLoopBackOff, ImagePullBackOff, and Terminating are kubectl display information and must not be confused with the Pod API's phase field.

Use STATUS to pick the next branch:

For phase versus STATUS rules, see Kubernetes Pods and Pod lifecycle.

Read conditions and events

kubectl describe pod exposes conditions such as PodScheduled, Initialized, Ready, and ContainersReady.

kubectl describe pod "$POD" -n pod-troubleshoot-lab | grep -A8 '^Conditions:'

Sample output:

Conditions:
  Type                        Status
  PodReadyToStartContainers   True 
  Initialized                 True 
  Ready                       False 
  ContainersReady             False 
  PodScheduled                True 
Volumes:

In this lab, PodScheduled=True, PodReadyToStartContainers=True, and Initialized=True show that scheduling, Pod sandbox setup, and initialization succeeded. ContainersReady=False and Ready=False then point to the regular application container.

Read warning events for the same Pod:

kubectl events -n pod-troubleshoot-lab --for "pod/$POD" --types=Warning

Sample output:

LAST SEEN           TYPE      REASON    OBJECT                           MESSAGE
13s (x2 over 25s)   Warning   BackOff   Pod/crash-web-5649dbcd64-4hj9q   Back-off restarting failed container app in pod crash-web-5649dbcd64-4hj9q_pod-troubleshoot-lab(9f760eb8-41d3-40b6-915f-1e3ce7a4b2ab)

Watch for these event reasons when you scan events:

• FailedScheduling
• FailedMount
• FailedCreatePodSandBox
• Failed with an image-pull error in the message
• BackOff
• Unhealthy

Flag syntax, sorting, and log flags are covered in kubectl logs, Events and describe.

Inspect Container Startup Failures

Current and previous logs

Logs show application output only after a container starts. They stay empty when the image never pulled or the container exited before writing to stdout.

Current logs from the crashing Pod:

kubectl logs "$POD" -n pod-troubleshoot-lab -c app

Sample output:

start

After a restart under restartPolicy: Always, read the previous instance:

kubectl logs "$POD" -n pod-troubleshoot-lab -c app --previous

Multi-container Pods need -c <name> or --all-containers=true. Init container logs use -c <init-container-name> before the main containers start.

If the container never ran, skip logs and read events and container state instead.

Image pull failures

When events mention ErrImagePull or the STATUS column shows ImagePullBackOff, inspect image configuration before you tune probes.

kubectl describe pod "$BAD" -n pod-troubleshoot-lab | grep -A8 '^  app:'

Sample output:

app:
    Container ID:   
    Image:          nginx:nonexistent-tag-xyz
    Image ID:       
    Port:           <none>
    Host Port:      <none>
    State:          Waiting
      Reason:       ImagePullBackOff

The useful pull failure reason is normally in events:

kubectl events -n pod-troubleshoot-lab --for "pod/$BAD" --types=Warning

Sample output:

TYPE      REASON    OBJECT                  MESSAGE
Warning   Failed    Pod/bad-image-...       Failed to pull image "nginx:nonexistent-tag-xyz": ...
Warning   Failed    Pod/bad-image-...       Error: ErrImagePull
Warning   BackOff   Pod/bad-image-...       Back-off pulling image "nginx:nonexistent-tag-xyz"
Warning   Failed    Pod/bad-image-...       Error: ImagePullBackOff

Keep the detailed runtime message trimmed because registry and container-runtime wording varies.

Command, arguments, and configuration

Check:

• Image name and tag (or digest)
• imagePullPolicy
• Container command and args
• ConfigMap and Secret references
• Required environment variables

Registry authentication and first-line pull errors are covered in Private registry image pulls. Command, args, and env patterns are in Kubernetes command, args and environment.

Container state, exit code, and OOMKilled

For crash loops, read current and last terminated state in describe:

kubectl describe pod "$POD" -n pod-troubleshoot-lab | grep -A12 'State:\|Last State:'

Sample output:

State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Sun, 26 Jul 2026 18:58:15 +0530
      Finished:     Sun, 26 Jul 2026 18:58:15 +0530

A stable crash loop has a current waiting state while the last completed instance retains its termination reason and exit code.

Note restart count, exit code, and termination reason together. Do not diagnose an out-of-memory failure from exit code 137 alone. Confirm that the termination reason is OOMKilled, then compare memory usage and limits. Kubernetes reports Reason: OOMKilled with exit code 137 when a container exceeds its memory limit—see OOMKilled and exit code 137. Repeated non-zero exits with BackOff events mean the process fails during startup; pair describe state with --previous logs.

Route Pending and NotReady Pods

Health probes

When phase is Running but READY stays 0/1, inspect probes before you change the image.

In describe, find Liveness, Readiness, and Startup probe blocks and match them to Unhealthy events. Wrong ports, paths, or timing produce readiness failures while the main process still runs.

Probe YAML, failure effects, and timing fields are documented in Kubernetes health probes.

Volumes, ConfigMaps, Secrets, and PVCs

Volume problems often appear as Pending, ContainerCreating, or FailedMount events before the main container starts.

Verify:

• PVC status and binding
• ConfigMap and Secret objects exist in the namespace
• volume and volumeMount name pairs match
• mountPath and optional subPath
• File permissions inside the mount
• FailedMount events on describe

Start with Kubernetes volumes for volume types and mount rules. PVC binding issues route to Kubernetes PersistentVolume and PVC. Multiple files in one directory may need Kubernetes subPath examples.

Scheduling and admission failures

When PodScheduled=False, the scheduler has not bound the Pod to a node. Review requests, affinity, taints, node state, and volume topology.

Review:

• CPU and memory requests versus node allocatable capacity
• nodeSelector, affinity, and anti-affinity rules
• Taints on nodes and tolerations on the Pod
• PVC binding and WaitForFirstConsumer
• Node Ready condition

With WaitForFirstConsumer, provisioning or binding waits until a Pod that uses the PVC exists, allowing the scheduler to consider the Pod's topology before selecting storage and a node.

ResourceQuota and LimitRange violations can reject object creation rather than leave a Pod waiting in Pending. WaitForFirstConsumer delays binding or provisioning so scheduling constraints can be considered together.

When a Deployment requests replicas but no Pod object exists, inspect the controller layer:

kubectl describe deployment <deployment> -n <namespace>

kubectl describe replicaset <replicaset> -n <namespace>

Kubernetes resources and requests and taints and tolerations cover the most common scheduler mismatches.

Check Service Routing After the Pod Is Ready

Service checks belong after the Pod reaches Ready. The lab crash-web Pod is not Ready, so its EndpointSlice entry illustrates readiness filtering rather than a targetPort defect.

Service selectors and ports

Confirm selector labels and ports:

kubectl get service crash-web -n pod-troubleshoot-lab -o jsonpath='selector={.spec.selector.app} port={.spec.ports[0].port} targetPort={.spec.ports[0].targetPort}{"\n"}'

Sample output:

selector=crash-web port=80 targetPort=80

EndpointSlice membership and readiness

The Endpoints API is deprecated starting with Kubernetes 1.33; use EndpointSlice for the tested Kubernetes 1.36 workflow.

kubectl get endpointslice -n pod-troubleshoot-lab -l kubernetes.io/service-name=crash-web -o jsonpath='{range .items[*].endpoints[*]}address={.addresses[0]}{" ready="}{.conditions.ready}{"\n"}{end}'

Sample output:

address=192.168.5.46 ready=false

The selector found the crashing Pod, so it appears in the EndpointSlice. Its endpoint is marked ready=false because the Pod is not Ready, and normal Service traffic does not use it. An empty EndpointSlice selection usually points to a selector mismatch or no matching Pods. A wrong numeric targetPort does not remove an otherwise Ready Pod from the EndpointSlice; it leaves an endpoint that sends traffic to a port where the application may not be listening.

EndpointSlice readiness follows the Pod Ready condition, except for cases such as publishNotReadyAddresses. A Service continuously selects matching Pods and maps its port to targetPort; endpoint membership and application listener correctness are separate checks.

DNS and application listener

Also verify:

• Application listens on the expected port and interface (not only 127.0.0.1)
• DNS name resolves inside the client Pod namespace

For a Ready Pod with an endpoint, verify that the application actually listens on the Service targetPort and on a reachable interface. A numeric targetPort does not require a matching containerPort declaration.

Service types, selectors, and ports are in Kubernetes Services. Name resolution problems often need Kubernetes DNS troubleshooting.

Pod Troubleshooting Decision Flow

Walk this sequence on every unhealthy Pod:

1. kubectl get pods -n <ns> -o wide
2. Inspect conditions, container state, and recent events with kubectl describe and kubectl events
3. Read current and --previous logs only when the container started
4. Check image, pull errors, command, arguments, environment, and configuration
5. Check probes when the process runs but the Pod is not Ready
6. Check mounts and PVC events for ContainerCreating
7. Check scheduling constraints when PodScheduled=False
8. Check Service selectors, EndpointSlices, ports, and DNS only after the Pod is Ready

On the lab crash-web Pod, steps 1–2 and container state pointed to exit code 1 and BackOff events before logs confirmed start. The bad-image Pod skipped logs and routed straight to image verification and pull events.

What's Next

• Fix Kubernetes CrashLoopBackOff Errors
• Fix Kubernetes ImagePullBackOff and ErrImagePull
• Fix Kubernetes Pods Stuck in Pending or ContainerCreating

References

• Debug Pods — upstream Pod debugging overview
• Determine Pod failure reason — exit codes and termination messages
• kubectl events — filter events by object

Summary

A reliable Pod troubleshoot starts with kubectl get for READY, STATUS, restarts, and node—not with random log tailing. Phase tells you the lifecycle bucket; STATUS routes you toward scheduling, image pull, crash loops, probes, or termination. Conditions and warning events narrow that branch within seconds.

The lab Pods showed both patterns: crash-web reached the node, started, exited with code 1, and emitted BackOff events; bad-image never pulled and stopped at ImagePullBackOff with pull errors in events rather than logs. EndpointSlice readiness matters when Pods look scheduled but clients still cannot connect.

This workflow deliberately stops at classification. Open the focused guide that matches your STATUS row instead of re-reading every fix on one page.