Fix Kubernetes CrashLoopBackOff Errors

Your Pod looks alive in the cluster but the container keeps dying. CrashLoopBackOff is the kubelet telling you it will try again, but not immediately. This walkthrough creates a deliberate crash in crash-lab, reads the signals that explain why the process exited, and applies the fix patterns that match real production failures.

Quick reference

Read termination state and previous logs before you change the image or command. Replace POD, NS, and CONTAINER with your values.

When STATUS begins with Init:, read init container logs with kubectl logs POD -n NS -c INIT_CONTAINER --previous before you tune the main application container.

What Does CrashLoopBackOff Mean?

A crash loop means a container repeatedly fails to start, exits, or is killed, and the kubelet retries it according to the applicable restart policy. Kubernetes describes CrashLoopBackOff as restart backoff after repeated container failures, including containers that fail to start properly.

After several failures the kubelet does not restart instantly every time. With default kubelet settings, restart delays begin at 10 seconds, double after repeated failures, and are capped at 5 minutes. The timer resets after the container runs successfully for 10 minutes. On Kubernetes 1.36, KubeletCrashLoopBackOffMax is beta and enabled by default. It allows each node to configure crashLoopBackOff.maxContainerRestartPeriod. The separate ReduceDefaultCrashLoopBackOffDecay feature remains alpha and disabled by default; when enabled, the default sequence starts at 1 second and is capped at 60 seconds instead of 10 seconds and 300 seconds.

The default sequence remains 10s, 20s, 40s up to 300s, with reset after approximately ten minutes of successful execution unless these settings are changed.

CrashLoopBackOff is a displayed container status reason, not a Pod phase. In kubectl describe, the failing container sits in Waiting with reason CrashLoopBackOff while it waits for the next restart. kubectl get may show CrashLoopBackOff, Error, or briefly Running as instances start and stop.

The Pod phase can stay Running because the Pod object is still bound to a node and the kubelet is actively managing containers. READY stays false when the main container is not ready. Read phase, STATUS, restart count, and container state together—not any single column alone.

Phase, STATUS, state, and restart backoff

Kubernetes tracks each container state as Waiting, Running, or Terminated. During a crash loop, the container normally waits in CrashLoopBackOff between attempts while the last failed execution remains under lastState.

Regular versus init-container crash loops

CrashLoopBackOff on a regular application container is different from Init:CrashLoopBackOff. When STATUS begins with Init:CrashLoopBackOff, an init container is repeatedly failing and the regular application containers may not have started. Inspect status.initContainerStatuses, then pass the init container name to kubectl logs.

kubectl get pod POD -n NAMESPACE -o jsonpath='{.status.initContainerStatuses}{"\n"}'

kubectl logs POD -n NAMESPACE -c INIT_CONTAINER --previous

Kubernetes reports init-container failures separately and supports retrieving their logs by container name.

Prepare and Confirm the CrashLoopBackOff Lab

Create and wait for the failing Pod

Create a Pod that prints one line and exits with code 1 on every start:

kubectl apply -f - <<'YAML'
apiVersion: v1
kind: Namespace
metadata:
  name: crash-lab
---
apiVersion: v1
kind: Pod
metadata:
  name: crash-exit1
  namespace: crash-lab
spec:
  containers:
  - name: app
    image: busybox:1.36
    command: ["sh", "-c", "echo crash-start; exit 1"]
YAML

Sample output:

namespace/crash-lab created
pod/crash-exit1 created

Wait for the stable backoff state:

kubectl wait pod/crash-exit1 -n crash-lab --for='jsonpath={.status.containerStatuses[0].state.waiting.reason}=CrashLoopBackOff' --timeout=120s

Sample output:

pod/crash-exit1 condition met

Representative status:

kubectl get pods -n crash-lab

Sample output:

NAME          READY   STATUS             RESTARTS   AGE
crash-exit1   0/1     CrashLoopBackOff   2          35s

Generated timing and restart counts vary with backoff timing. CrashLoopBackOff means the kubelet is currently delaying another restart after repeated failures.

Read state, restart count, and Events

Confirm phase separately because STATUS is only a summary:

kubectl get pod crash-exit1 -n crash-lab -o jsonpath='{.status.phase}{"\n"}'

Sample output:

Running

Phase Running with STATUS CrashLoopBackOff is normal for a crash loop. The Pod is on a node; the application container is not healthy.

Open the full report for container state and events:

kubectl describe pod crash-exit1 -n crash-lab

Scroll to the container block. Once the Pod is in stable CrashLoopBackOff, the current state is normally Waiting, while the previous instance appears under Last State: Terminated:

State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
    Restart Count:  2

Do not describe both states as terminated. A crash-looping container waits during the backoff interval while its last failed execution remains under lastState.

Pull only the Events tail when you already know the Pod name:

kubectl describe pod crash-exit1 -n crash-lab | grep -A8 '^Events:'

Sample output:

Events:
  Type     Reason     Age               From               Message
  ----     ------     ----              ----               -------
  Normal   Scheduled  45s               default-scheduler  Successfully assigned crash-lab/crash-exit1 to worker01
  Normal   Pulled     15s (x3 over 45s)  kubelet            Container image "busybox:1.36" already present on machine
  Normal   Created    15s (x3 over 45s)  kubelet            Container created
  Normal   Started    15s (x3 over 45s)  kubelet            Container started
  Warning  BackOff    5s (x4 over 43s)   kubelet            Back-off restarting failed container app in pod crash-exit1_crash-lab(...)

The exact BackOff aggregation count can vary. Under default settings, restart delays start at 10 seconds and double between failed attempts.

Warning events with reason BackOff confirm the kubelet is throttling restarts after repeated failures—not an image pull or scheduling problem.

Compare current and previous logs

kubectl logs reads the latest container instance. --previous reads the instance immediately before it, when one exists. During a crash loop, compare both because the newest attempt may have different output or may have failed before logging.

--previous specifically requests logs from the previous container instance; it is not simply a fallback whenever the latest instance has terminated.

kubectl logs crash-exit1 -n crash-lab -c app

kubectl logs crash-exit1 -n crash-lab -c app --previous

Sample output:

crash-start

That line proves the script ran and exited. For multi-container Pods, add -c <container-name> so you read the failing container, not a sibling.

Inspect termination reason and exit code

kubectl describe and kubectl get pod -o jsonpath expose the last termination reason and exit code. Treat the code as a hint—confirm with logs and events.

Read the exit code from the lab Pod:

kubectl get pod crash-exit1 -n crash-lab -o jsonpath='{.status.containerStatuses[0].lastState.terminated.exitCode}{"\n"}'

Sample output:

1

Exit code 1 matches the intentional exit 1 in the lab script and the Error termination reason in describe.

Fix Container Startup Failures

Incorrect command or arguments

Wrong command, args, or image entrypoint combinations are a frequent crash-loop cause. The container starts, the shell or runtime cannot run what you asked for, and the process exits immediately.

Check these layers together:

• Image ENTRYPOINT and CMD from the image metadata
• Kubernetes command and args overrides in the Pod spec
• Shell availability (sh or bash present in minimal images)
• Quoting and path spelling in chained shell commands
• Whether the binary path is absolute or on PATH

A directly configured missing executable can produce StartError before the application process begins. If a shell starts successfully but cannot find a command inside the script, the shell commonly exits with 127. Example from a Pod with command: ["missing-binary"]:

Reason:       StartError
Message:      ... exec: "missing-binary": executable file not found in $PATH
Exit Code:    128

Fix by correcting the command, using an image that contains the binary, or wrapping the call in a shell that exists. Full override behavior is covered in Commands, args and environment variables.

Missing or invalid configuration

Missing or wrong configuration often surfaces as a crash loop when the application reads config at startup and exits on failure.

Check:

• ConfigMap and Secret names referenced in volumes or env vars
• Key names inside those objects
• configMapKeyRef / secretKeyRef name and key fields
• Mount paths and whether files exist where the app expects them
• Application syntax for the config format (YAML, JSON, properties)

ConfigMap and Secret references in a Pod must resolve to objects in the same namespace as that Pod.

Distinguish the resulting states:

• Missing required ConfigMap or Secret volume → commonly FailedMount or configuration startup failure before the application runs
• Missing required key or environment reference → commonly CreateContainerConfigError
• Present but invalid application configuration → container may start, log an error, exit, and eventually reach CrashLoopBackOff

Only the last category is a true application crash loop.

Example from a missing ConfigMap volume:

Warning  FailedMount  ...  kubelet  MountVolume.SetUp failed for volume "cfg" : configmap "app-config-missing" not found

That is not CrashLoopBackOff yet—the kubelet never started the app container. Create the ConfigMap or fix the volume reference first.

After correcting a ConfigMap or Secret, recreate or restart the workload when the value is injected through environment variables or the application reads configuration only during startup. For controller-owned workloads, update or restart the controller rather than repeatedly deleting an unchanged Pod.

Permission and filesystem failures

Permission failures often appear in logs as permission denied or as exit code 126. The process starts, cannot read a mounted file or write a required directory, and exits.

Check:

• Executable bit on scripts referenced in command
• Ownership and mode on mounted volumes (NFS, hostPath, PVC)
• Read-only root filesystem or read-only volume mounts
• Container user (runAsUser, runAsNonRoot) versus file ownership on the volume
• Writable paths the application needs (/tmp, cache dirs, pid files)

This article does not walk through SecurityContext design. When describe and logs point at filesystem errors, fix mount permissions or align the runtime user with volume ownership, then confirm the process can write before expecting the Pod to become Ready.

Fix Kubelet-Triggered Restarts

Liveness and startup probes

Liveness and startup probe failures restart the container. When probes are wrong, events mention Unhealthy and Liveness probe failed rather than a clean application exit. Example from nginx with a bad HTTP liveness path:

Warning  Unhealthy  ...  kubelet  Liveness probe failed: HTTP probe failed with statuscode: 404
Normal   Killing    ...  kubelet  Container app failed liveness probe, will be restarted

Common probe mistakes:

• HTTP path does not exist (/healthz vs /health)
• Probe targets the wrong port or named port
• Probe runs before the application listens—tune initialDelaySeconds or add a startup probe
• timeoutSeconds too short for slow handlers
• Application listens only on 127.0.0.1 while the probe expects the pod network interface

For probe YAML, timing fields, and startup versus liveness behavior, see Kubernetes health probes.

OOMKilled containers

When memory exceeds the container limit, the kernel kills the process. Describe shows reason OOMKilled and exit code 137 (128 + signal 9). Do not diagnose OOM from 137 alone; the termination reason must confirm OOMKilled.

Inspect:

• lastState.terminated.reason and exitCode in describe
• resources.limits.memory on the container
• Current usage with metrics tooling when available
• Application memory growth during startup spikes

This article routes deep OOM workflow to OOMKilled and exit code 137. Here the signal is enough to know the crash is memory-related rather than a bad command or probe.

Readiness failures that do not restart containers

Readiness failure marks the Pod NotReady. A matching EndpointSlice can retain its address with ready=false, but normal Service traffic will not be sent to it. EndpointSlice readiness explicitly indicates whether an endpoint should receive traffic.

Readiness failure alone leaves STATUS as Running with reduced READY count—clients may see connection failures without restarts climbing.

Fix Application Dependency Failures

Some applications exit immediately when a database, API, message broker, or peer Service is unreachable at boot. Kubernetes does not retry application-level dependency checks for you—the kubelet only restarts the container.

Check:

• Service name and namespace in connection URLs
• DNS resolution for cluster Services (my-svc.my-ns.svc.cluster.local)
• NetworkPolicy or firewall rules blocking egress
• Init containers that must complete before application containers start
• Setup Jobs that must finish before the application becomes usable
• Companion or sidecar containers whose runtime service must remain available

Kubernetes-native sidecars restart independently and run through the Pod lifetime rather than acting as completion gates.

Fix the endpoint, add application retry logic for transient outages, or use an init container to gate startup when the dependency must exist first. Rising restarts with dependency errors in --previous logs often clear once the upstream Service has endpoints.

Debug a Container That Exits Too Quickly

When the container exits faster than kubectl exec allows, copy the Pod and override the command so you can inspect files and run the entrypoint manually. Debug Pods with kubectl exec and debug covers ephemeral containers and copy workflows in full.

Copy the Pod with a stable command

Command override with --container=app works when the original image contains the replacement command. The BusyBox lab image contains sh, so this pattern is valid:

kubectl debug crash-exit1 -n crash-lab --copy-to=crash-exit1-debug --container=app -- sh -c 'sleep 600'

Wait for the copied Pod:

kubectl wait pod/crash-exit1-debug -n crash-lab --for=condition=Ready --timeout=120s

Sample output:

pod/crash-exit1-debug condition met

The copy is a standalone Pod and does not replace or modify the original crash-exit1 Pod:

kubectl get pod crash-exit1 crash-exit1-debug -n crash-lab

Open a shell in the stable copy:

kubectl exec -it crash-exit1-debug -n crash-lab -c app -- sh

From the debug copy you can list mounted config paths and run the real command by hand to capture stderr before the next restart.

Use another image when the application image lacks tools

When the image lacks a shell or debugging tools, use a copied Pod with a different debug image or add a dedicated debug container. The official command supports both changing an existing container command and changing container images with --set-image.

Verify the Fix

After you change the Deployment, Pod template, or underlying ConfigMap, confirm the failure stopped—not only that one manual kubectl run worked.

Check:

• RESTARTS on kubectl get stops increasing over several minutes
• Pod reaches expected READY count (1/1 for a single-container Pod)
• Events no longer show BackOff, Unhealthy, or repeated Killing for the same reason
• kubectl logs and --previous show normal startup when you force another restart
• Controller rollout completes—kubectl rollout status deployment/<name> when a Deployment owns the Pod

Standalone Pod replacement

Container commands are part of the Pod specification and cannot be updated in place on a standalone Pod. Create a replacement Pod for the lab. For a Deployment, StatefulSet, or DaemonSet, update the controller's Pod template so it creates replacement Pods.

kubectl apply -f - <<'YAML'
apiVersion: v1
kind: Pod
metadata:
  name: crash-fixed
  namespace: crash-lab
spec:
  containers:
  - name: app
    image: busybox:1.36
    command: ["sh", "-c", "echo crash-start; sleep 3600"]
YAML

kubectl wait pod/crash-fixed -n crash-lab --for=condition=Ready --timeout=120s

Sample output:

pod/crash-fixed condition met

kubectl get pod crash-fixed -n crash-lab

Sample output:

NAME          READY   STATUS    RESTARTS   AGE
crash-fixed   1/1     Running   0          12s

The replacement Pod using the same image stays running because its command no longer exits immediately. Zero restarts and 1/1 Ready is the baseline signal that the kubelet is no longer cycling the container.

Controller-owned workload rollout

For controller-owned Pods, verify the new ReplicaSet Pods reach Ready and that kubectl rollout status completes before you treat the crash loop as fixed.

Troubleshoot Common CrashLoopBackOff Patterns

What's Next

• Fix Kubernetes ImagePullBackOff and ErrImagePull
• Fix Kubernetes Pods Stuck in Pending or ContainerCreating
• Fix Kubernetes OOMKilled and Exit Code 137

References

• Determine the reason for Pod failure — exit codes and termination messages
• Debug Pods — exec, ephemeral containers, and copy-to debugging
• Configure Liveness, Readiness and Startup Probes — upstream probe configuration

Summary

CrashLoopBackOff means the kubelet is retrying a container that keeps exiting or getting killed, with increasing delay between attempts. It is a container Waiting reason and a kubectl STATUS hint—not a Pod phase. You confirmed the pattern in crash-lab with climbing RESTARTS, phase Running, stable Waiting state with CrashLoopBackOff, BackOff events, exit code 1, and crash-start in the logs.

The fix path follows the termination signal: bad commands and missing binaries show up in exit codes and StartError messages; missing mounts stall in ContainerCreating; probe failures appear as Unhealthy and Killing events; OOM kills require OOMKilled with exit 137. Readiness failures alone do not restart containers—do not tune liveness when only Service traffic is wrong.

Once you identify the category, apply the matching correction and verify that restarts stop and Ready becomes true. For standalone Pods, create a replacement; for controllers, update the Pod template. When the process exits too fast for exec, copy the Pod with kubectl debug --copy-to and inspect from a stable shell. For STATUS rows outside the crash loop—image pull, scheduling, or Service routing—return to the Pod troubleshooting workflow instead of repeating crash-loop checks.