Fix FailedCreatePodSandBox and Kubernetes CNI Errors

FailedCreatePodSandBox means the kubelet asked the runtime to create the Pod network sandbox and that step failed. The Pod is usually already scheduled, so you are debugging a node networking problem—not the scheduler. This walkthrough starts from the Event text, decides whether the blast radius is one node or the whole cluster, then checks CNI agents, configuration, binaries, and runtime logs.

What happens during Pod sandbox creation?

After the scheduler picks a node, the kubelet asks the CRI runtime to create a Pod sandbox. The runtime then invokes CNI plugins to attach networking. Only after that succeed do application containers start.

Pod scheduled to node
  ↓
kubelet requests Pod sandbox through CRI
  ↓
runtime creates sandbox
  ↓
runtime invokes CNI plugins
  ↓
Pod receives networking
  ↓
application containers start

A failure before the last step commonly leaves the Pod in Pending or ContainerCreating with no useful application logs. Related Events include FailedCreatePodSandBox and sometimes NetworkNotReady when the node itself reports that CNI is not ready.

For the broader Pending versus ContainerCreating decision tree, see Pending and ContainerCreating Pods.

Confirm the exact Event

Describe the stuck Pod and capture the full message after FailedCreatePodSandBox or NetworkNotReady. Replace the namespace and Pod name with yours.

kubectl -n sandbox-lab describe pod sandbox-fail2 | sed -n '/Events:/,$p'

Events:
  Type     Reason                  Age   From     Message
  ----     ------                  ----  ----     -------
  Warning  FailedCreatePodSandBox  3s    kubelet  Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "e2aea0e3d5a9089636a0160482cfa99f85588b46d7e340a8597210971e1066fa": plugin type="calico" failed (add): failed to find plugin "calico" in path [/opt/cni/bin]

The useful part is after the reason name. Here the runtime found a CNI config that calls plugin type calico, but the binary was missing from /opt/cni/bin.

List Warning Events for the same object:

kubectl events -n sandbox-lab --for pod/sandbox-fail2 --types=Warning

LAST SEEN   TYPE      REASON                   OBJECT              MESSAGE
4s          Warning   FailedCreatePodSandBox   Pod/sandbox-fail2   Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "e2aea0e3d5a9089636a0160482cfa99f85588b46d7e340a8597210971e1066fa": plugin type="calico" failed (add): failed to find plugin "calico" in path [/opt/cni/bin]

When CNI config is missing entirely, you may see NetworkNotReady instead of—or before—FailedCreatePodSandBox:

Warning  NetworkNotReady  1s (x21 over 41s)  kubelet  network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized

Application logs are normally empty at this stage because containers have not started. Stay on Events and node logs.

Determine the failure scope

Decide whether you are chasing one bad Pod, one bad node, or a cluster-wide CNI outage.

List Pods with nodes so concentration is obvious:

kubectl get pods -A -o wide | head -20

Compare the NODE column for every ContainerCreating Pod. If they all land on worker01, start on that node. If every node shows the same Event text, check whether CNI was installed cluster-wide.

Also check whether the assigned node is still Ready:

kubectl get nodes

NAME       STATUS   ROLES           AGE    VERSION
k8s-cp     Ready    control-plane   117m   v1.36.3
worker01   Ready    <none>          115m   v1.36.3

When CNI config disappears, the same node can flip to NotReady with KubeletNotReady and cni plugin not initialized. That is a node-scope signal even if only one new Pod is stuck.

Check node and CNI component status

Confirm nodes and look for CNI DaemonSets that are not Ready on the affected node.

kubectl get daemonsets -A | grep -Ei 'calico|cilium|flannel|weave|cni'

calico-system   calico-node       2         2         2       2            2           kubernetes.io/os=linux   114m
calico-system   csi-node-driver   2         2         2       2            2           kubernetes.io/os=linux   114m

On this lab Calico’s node agent is calico-node. Your cluster may use a different DaemonSet name.

Inspect CNI Pods with node placement:

kubectl -n calico-system get pods -o wide

NAME                                       READY   STATUS    RESTARTS      AGE    IP               NODE       NOMINATED NODE   READINESS GATES
calico-apiserver-59bcc9778f-2gsfr          1/1     Running   0             101m   192.168.62.146   k8s-cp     <none>           <none>
calico-apiserver-59bcc9778f-xtqp6          1/1     Running   0             103m   192.168.62.141   k8s-cp     <none>           <none>
calico-kube-controllers-65898666fc-9nng8   1/1     Running   0             103m   192.168.62.140   k8s-cp     <none>           <none>
calico-node-94m77                          1/1     Running   0             114m   192.168.56.108   k8s-cp     <none>           <none>
calico-node-j9m4s                          1/1     Running   0             113m   192.168.56.109   worker01   <none>           <none>
calico-typha-7bf79c9c5d-pczs2              1/1     Running   0             103m   192.168.56.108   k8s-cp     <none>           <none>
csi-node-driver-grh5c                      2/2     Running   0             114m   192.168.62.135   k8s-cp     <none>           <none>
csi-node-driver-hczmm                      2/2     Running   0             113m   192.168.5.0      worker01   <none>           <none>

Look for:

• No CNI DaemonSet Pod on the affected node
• CrashLoopBackOff or Pending CNI agents
• Missing controller Pods for your plugin family
• Node NotReady while CNI Pods restart

If the DaemonSet never scheduled a Pod on a new worker, the node often stays NotReady until CNI writes /etc/cni/net.d.

Inspect kubelet and runtime logs

SSH to the node named in the Pod’s NODE column. Use the runtime your Node reports (containerd here). Do not assume Docker.

Confirm the runtime kubelet is talking to:

crictl info | python3 -c 'import sys,json; d=json.load(sys.stdin); print(json.dumps(d.get("config",{}).get("cni",{}), indent=2))'

{
  "binDir": "",
  "binDirs": [
    "/opt/cni/bin"
  ],
  "confDir": "/etc/cni/net.d",
  "confTemplate": "",
  "ipPref": "",
  "maxConfNum": 1,
  "setupSerially": false,
  "useInternalLoopback": false
}

This lab watches /etc/cni/net.d and /opt/cni/bin. If those paths differ in your runtime config, inspect the directories the runtime actually uses.

Read recent kubelet messages for the Pod:

journalctl -u kubelet --since "5 min ago" --no-pager | grep -iE 'cni|NetworkPlugin|sandbox|FailedCreatePodSandBox' | tail -10

E0727 20:06:58.789242  706382 pod_workers.go:1338] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized" pod="sandbox-lab/sandbox-fail" podUID="faddcd29-59c0-4e32-bc41-2b932711330c"
I0727 20:06:58.985360  706382 kubelet.go:3262] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized"

Then check the runtime for the CNI ADD/DELETE error:

journalctl -u containerd --since "5 min ago" --no-pager | grep -iE 'cni|sandbox|plugin' | tail -10

level=error msg="failed to reload cni configuration after receiving fs change event(RENAME        \"/etc/cni/net.d/10-calico.conflist\")" error="cni config load failed: no network config found in /etc/cni/net.d: cni plugin not initialized: failed to load cni config"
level=error msg="RunPodSandbox for name:\"sandbox-fail2\" ... failed" error="rpc error: code = Unknown desc = failed to setup network for sandbox \"...\": plugin type=\"calico\" failed (add): failed to find plugin \"calico\" in path [/opt/cni/bin]"

Those two messages map cleanly to fixes: restore a valid config under confDir, or restore the missing plugin binary under binDirs.

Inspect CNI configuration and binaries

On the affected node, list the CNI config directory the runtime uses:

ls -la /etc/cni/net.d/

Healthy Calico lab output:

total 16
drwx------. 2 root root 4096 Jul 27 18:11 .
drwxr-xr-x. 3 root root 4096 Jul 27 18:08 ..
-rw-------. 1 root root  735 Jul 27 18:11 10-calico.conflist
-rw-------. 1 root root 2801 Jul 27 19:57 calico-kubeconfig

When the conflist is removed, only credentials may remain and the runtime reports no network config found:

total 12
drwx------. 2 root root 4096 Jul 27 20:06 .
drwxr-xr-x. 3 root root 4096 Jul 27 18:07 ..
-rw-------. 1 root root 2801 Jul 27 20:03 calico-kubeconfig

Check binaries next:

ls -l /opt/cni/bin/ | head -20

-rwxr-xr-x. 1 root root  ... bandwidth
-rwxr-xr-x. 1 root root  ... bridge
-rwxr-xr-x. 1 root root  ... calico
-rwxr-xr-x. 1 root root  ... calico-ipam
-rwxr-xr-x. 1 root root  ... host-local
-rwxr-xr-x. 1 root root  ... loopback
-rwxr-xr-x. 1 root root  ... portmap

What to look for:

• Empty /etc/cni/net.d or no .conf / .conflist
• Multiple primary configs fighting for maxConfNum (often the first lexicographic file wins)
• Invalid JSON
• Plugin type that does not match a file under /opt/cni/bin
• Missing binary (this lab’s failed to find plugin "calico" case)
• Unsupported or undeclared CNI version for your runtime
• Runtime confDir / binDirs pointing somewhere else

Fix NetworkPluginNotReady

NetworkPluginNotReady with cni plugin not initialized means the runtime has no usable CNI configuration yet. Work through these checks in order:

1. Confirm a CNI was installed for the cluster (install Kubernetes with kubeadm covers the kubeadm CNI step).
2. Confirm the node agent DaemonSet is Ready on this node.
3. Confirm a .conf or .conflist exists under the runtime confDir.
4. Confirm the Pod CIDR the control plane expects matches the plugin pool.
5. Restart the runtime only after required config is present, if the runtime did not reload automatically.
6. Confirm kubeadm init used a compatible --pod-network-cidr for your plugin.

On this kubeadm lab the Pod subnet is:

kubectl -n kube-system get cm kubeadm-config -o yaml | grep -A5 'networking:'

networking:
      dnsDomain: cluster.local
      podSubnet: 192.168.0.0/16
      serviceSubnet: 10.96.0.0/12

After restoring /etc/cni/net.d/10-calico.conflist on the broken worker, the node returned to Ready without a reinstall:

kubectl get nodes

NAME       STATUS   ROLES           AGE    VERSION
k8s-cp     Ready    control-plane   117m   v1.36.3
worker01   Ready    <none>          115m   v1.36.3

If the directory stays empty because the CNI agent never runs, fix the DaemonSet first—do not invent a one-off conflist that fights your installed plugin.

Fix IPAM and address exhaustion

Only dig into IPAM after the Event mentions allocation failure, pool exhaustion, or “no available addresses.” Generic cni plugin not initialized is usually config or agent readiness, not IPAM.

Calico example (this lab)

List Calico IP pools when Events point at allocation:

kubectl get ippools.crd.projectcalico.org

NAME                  AGE
default-ipv4-ippool   119m

Inspect the CIDR and compare it with podSubnet:

kubectl get ippools.crd.projectcalico.org default-ipv4-ippool -o yaml | sed -n '/^spec:/,/^status:/p'

spec:
  allowedUses:
  - Workload
  - Tunnel
  assignmentMode: Automatic
  blockSize: 26
  cidr: 192.168.0.0/16
  ipipMode: Never
  natOutgoing: true
  nodeSelector: all()
  vxlanMode: CrossSubnet

Here cidr: 192.168.0.0/16 matches kubeadm podSubnet. A mismatch is a common reason plugins refuse to initialize or fail ADD.

Other plugins expose pools differently (Cilium IPAM CRDs, host-local ranges in the conflist, cloud VPC limits). Stay on the Event text: if it never mentions IP allocation, finish config and agent checks first.

Fix route, interface, and dataplane failures

When Events or CNI logs mention routes, interfaces, MTU, tunnels, or iptables/nftables, check only the signals that match the message:

• Expected host interface for the plugin is present (ip link)
• Node has routes for the Pod CIDR your plugin programs
• MTU mismatches between tunnel and underlay
• Required kernel modules for the chosen dataplane
• Host firewall blocking node-to-node overlay or BGP
• Overlapping node, Pod, and Service ranges

Do not turn this into a full Linux networking rebuild. If Pods get IPs but cannot talk across nodes, continue in Kubernetes networking after sandbox creation succeeds.

Use crictl for sandbox inspection

On the affected node, list sandboxes the runtime knows about:

crictl pods

POD ID              CREATED             STATE               NAME                NAMESPACE           ATTEMPT             RUNTIME
db6e3600bdb15       3 minutes ago       Ready               healthy-echo        sandbox-lab         0                   (default)

A failed attempt may leave a short-lived NotReady sandbox, or may fail before a lasting sandbox ID remains. When you have an ID, inspect it:

crictl inspectp db6e3600bdb1518151fc8e74b2e09cfd9e87cfda8b009c59691f8bab712eee2c | python3 -c 'import sys,json; s=json.load(sys.stdin).get("status",{}); print("state", s.get("state")); print("network", s.get("network")); print("reason", s.get("reason")); print("message", s.get("message"))'

state SANDBOX_READY
network {'additionalIps': [], 'ip': '192.168.5.34'}
reason None
message None

A healthy sandbox shows SANDBOX_READY and a Pod IP. For failures, lean on the kubelet Event and journalctl -u containerd lines for RunPodSandbox—those often retain the CNI ADD error after the sandbox object is gone.

Node service logs on systemd hosts use journalctl filtering.

Verify the fix

After you restore config or binaries (or heal the CNI agent), confirm the full path works again.

1. CNI agent Ready on the node:

kubectl -n calico-system get pods -l k8s-app=calico-node -o wide

NAME                READY   STATUS    RESTARTS   AGE    IP               NODE       NOMINATED NODE   READINESS GATES
calico-node-94m77   1/1     Running   0          118m   192.168.56.108   k8s-cp     <none>           <none>
calico-node-j9m4s   1/1     Running   0          118m   192.168.56.109   worker01   <none>           <none>

2. Create a new test Pod on the previously broken node and wait until it is Running with an IP:

kubectl -n sandbox-lab get pods -o wide

NAME           READY   STATUS    RESTARTS   AGE    IP             NODE       NOMINATED NODE   READINESS GATES
healthy-echo   1/1     Running   0          4m4s   192.168.5.34   worker01   <none>           <none>
sandbox-ok     1/1     Running   0          3s     192.168.5.29   worker01   <none>           <none>

3. Pod-to-Pod traffic on the node (or across nodes when both are healthy):

kubectl -n sandbox-lab exec netcheck -- ping -c 2 192.168.5.34

PING 192.168.5.34 (192.168.5.34): 56 data bytes
64 bytes from 192.168.5.34: seq=0 ttl=42 time=5.175 ms
64 bytes from 192.168.5.34: seq=1 ttl=42 time=0.141 ms

--- 192.168.5.34 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss

4. Service traffic once backends are Running. Prefer ClusterIP while DNS recovers after a node CNI outage:

kubectl -n sandbox-lab get svc web -o jsonpath='{.spec.clusterIP}{"\n"}'

10.108.106.113

kubectl -n sandbox-lab run svccheck2 --rm -i --restart=Never --image=curlimages/curl:8.5.0 -- curl -sS -o /dev/null -w "%{http_code}\n" --connect-timeout 5 http://10.108.106.113/

200

5. Create a second test Pod and confirm FailedCreatePodSandBox does not return.

If Pods run and ping works but Services still fail, leave this article and use Troubleshoot a Service that is not working.

Common error routing

What's Next

• Monitor Kubernetes Pods and Nodes with kubectl top
• kubectl logs, Events and describe with Examples
• Debug Kubernetes Pods with kubectl exec and kubectl debug

References

• Troubleshooting CNI plugin-related errors (kubernetes.io)
• Cluster networking (kubernetes.io)
• CNI specification
• containerd documentation
• Calico documentation

Summary

FailedCreatePodSandBox is a node-local networking failure after scheduling. Read the full Event, decide whether one node or the whole cluster is affected, then verify the CNI DaemonSet, /etc/cni/net.d, /opt/cni/bin, and runtime logs before chasing IPAM or routes.

On this lab, removing the Calico conflist produced NetworkPluginNotReady / cni plugin not initialized, and removing the calico binary produced FailedCreatePodSandBox with failed to find plugin "calico". Restoring the file or binary brought the node and new Pods back without reinstalling Kubernetes.

When sandbox creation succeeds, confirm a Pod IP, Pod-to-Pod reachability, and a Service ClusterIP check. If those pass and users still report DNS or Service symptoms, that is a different problem path than CNI sandbox setup.