| Tested on | Rocky Linux 10.2 (Red Quartz) workstation |
|---|---|
| Package | kubectl 1.36.3containerd 2.2.5crictl v1.36.0 |
| Applies to | Ubuntu, Debian, Kali Linux, Linux Mint, Pop!_OS, Raspberry Pi OS, elementary OS, Zorin OS, Parrot OS, MX Linux, RHEL, Rocky Linux, AlmaLinux, Oracle Linux, CentOS Stream, Fedora Kubernetes cluster |
| Cert prep | CKA |
| Lab environment | Multi-node kubeadm cluster with containerd — install Kubernetes with kubeadm |
| Privilege | Normal user for kubectl; sudo on cluster nodes for crictl and CNI directory inspection |
| Kubernetes permissions | Get/list Nodes, Pods, DaemonSets, CSIDrivers, CSINodes, StorageClasses, and VolumeAttachments; get nodes/proxy only when using the API-server proxy command for kubelet configz |
| Additional tool | Python 3 for parsing the kubelet configz response |
| Scope | Architecture comparison of CRI, CNI, and CSI, discovery commands for runtime, network plugin, and storage drivers, and a symptom routing table. Does not cover installing runtimes, CNI plugins, or CSI drivers, packet-level networking, kube-proxy modes, or StorageClass authoring. |
Kubernetes keeps its core small by defining extension interfaces instead of shipping every container runtime, network plugin, and storage backend. When a Pod lands on a node, the kubelet talks to a CRI runtime, the runtime invokes CNI to wire the sandbox network, and CSI components handle volume operations when a Pod needs CSI-backed storage. This guide maps those boundaries and shows commands that reveal what your cluster actually runs.
Why Kubernetes uses extension interfaces
Upstream Kubernetes focuses on scheduling, API objects, and reconciliation. Container execution, Pod networking, and block or file storage vary by distribution, cloud, and hardware. Rather than embed one implementation, Kubernetes integrates with pluggable interfaces. CRI defines the kubelet-to-runtime contract, while Kubernetes adopts the CNI networking and CSI storage specifications for external implementations.
CNI is maintained as a separate CNCF interface specification, while CSI drivers normally expose separate controller and node services.
On a worker node the relationships look like this:
kubelet
├── CRI → container runtime
│ └── CNI → Pod network plugins
└── CSI node service → node mount operations
Kubernetes controllers and CSI sidecars
└── CSI controller service → provision, attach, resize, and snapshot operationsCRI, CNI, and CSI are specifications — not products you install by those names alone. You choose a CRI-compatible runtime (containerd or CRI-O), a CNI implementation (Calico, Cilium, Flannel, and others), and CSI drivers that match your storage needs. The Kubernetes architecture guide covers how those pieces sit beside the control plane; here the focus is what each interface standardizes and how to identify the active implementation.
CNI vs CSI vs CRI
The three interfaces answer different questions on the node:
| Interface | Standardizes | Typical implementations |
|---|---|---|
| CRI | kubelet-to-container-runtime communication | containerd, CRI-O |
| CNI | Pod network-interface configuration | Calico, Cilium, Flannel |
| CSI | Storage provisioning, attachment, and mounting | Cloud, SAN, platform, or ephemeral CSI drivers |
CRI handles image pulls and container lifecycle. CNI runs when the runtime creates a Pod sandbox and assigns an IP address. CSI connects volume operations to external drivers — including persistent PVC-backed volumes and inline ephemeral volumes. They stack rather than replace one another — a storage failure does not look like a CNI misconfiguration, and confusing the three slows troubleshooting.
containerd:// or cri-o:// in node status as the runtime signal.
How CRI connects kubelet to the runtime
The kubelet on each node is the agent that starts Pods. It dials a configured CRI endpoint — commonly a Unix socket such as unix:///var/run/containerd/containerd.sock — and uses two gRPC services:
- Runtime service — create Pod sandboxes, start and stop containers, report status
- Image service — pull, list, and remove images
When the socket path is wrong or the runtime is down, the node may stay NotReady even though the API server is healthy. That failure boundary is distinct from a Pod that schedules but cannot get an IP.
Read the runtime from the Node object
Start from the API. The Node status field reports the runtime the kubelet registered:
kubectl get node worker01 -o jsonpath='{.status.nodeInfo.containerRuntimeVersion}{"\n"}'Sample output:
containerd://2.2.5The containerd:// prefix names the CRI-compatible runtime; 2.2.5 is the runtime version on that node.
Read the kubelet CRI endpoint
When you need the exact socket path kubelet was configured with, query the kubelet configuration through the API. This command can return Forbidden unless your account may access the kubelet configz endpoint.
kubectl get --raw /api/v1/nodes/worker01/proxy/configz | python3 -c "import sys,json; k=json.load(sys.stdin)['kubeletconfig']; print(k.get('containerRuntimeEndpoint',''))"Sample output:
unix:///var/run/containerd/containerd.sockThe API-server proxy URL requires get permission on nodes/proxy, which is broader than ordinary Node read access. Kubernetes 1.36 additionally uses nodes/configz when the kubelet authorizes the identity connecting to its /configz endpoint. Use the node-local kubeadm file below when your kubectl account does not have nodes/proxy.
For kubeadm clusters, a node-local alternative does not need the Node proxy API:
sudo grep '^containerRuntimeEndpoint:' /var/lib/kubelet/instance-config.yamlcontainerRuntimeEndpoint: "unix:///var/run/containerd/containerd.sock"Kubeadm records the detected CRI socket in /var/lib/kubelet/instance-config.yaml. That endpoint must exist and be reachable. If kubelet cannot connect, new Pods will not start on the node.
Confirm the CRI socket and runtime with crictl
On the node, crictl speaks the same CRI API the kubelet uses. Pass the runtime endpoint explicitly so you do not depend on /etc/crictl.yaml or automatic probing when multiple sockets exist:
sudo crictl --runtime-endpoint unix:///var/run/containerd/containerd.sock versionSample output:
Version: 0.1.0
RuntimeName: containerd
RuntimeVersion: 2.2.5
RuntimeApiVersion: v1crictl version queries the runtime’s CRI version information. Version: 0.1.0 is the legacy kubelet runtime API version string returned in the CRI VersionResponse. Use RuntimeApiVersion: v1 to confirm that the runtime supports the required CRI v1 API. To display the installed crictl client version, run:
crictl --versioncrictl version v1.36.0The runtime name and version should align with the Node object. A mismatch between what the API reports and what crictl reaches usually means you are inspecting a different socket than kubelet uses.
You can also dump runtime configuration against the same endpoint:
sudo crictl --runtime-endpoint unix:///var/run/containerd/containerd.sock infoSample output (trimmed):
{
"cniconfig": {
"Networks": [
{
"Config": {
"CNIVersion": "0.3.1",
"Name": "cni-loopback",
...The cniconfig section shows how the runtime loads CNI definitions — a reminder that CRI and CNI cooperate during sandbox creation even though they are separate specifications.
How CNI connects Pods to the network
Kubernetes assumes every Pod gets a routable IP and that Pods on different nodes can reach each other without manual NAT rules. The project does not implement that routing itself. Instead, the container runtime calls CNI plugins listed in configuration files on the node.
Typical layout on Linux:
- Network definitions —
/etc/cni/net.d/(.conflistor.conffiles) - Plugin binaries —
/opt/cni/bin/
Containerd defaults to those directories and to one loaded CNI configuration, but the values are configurable in the runtime. A CNI product often ships both the plugin binaries and cluster-level controllers. Calico, for example, runs calico-node DaemonSet pods that program routes and policy while the runtime invokes the calico binary during sandbox setup. NetworkPolicy support depends on the implementation — not every CNI plugin enforces policy.
This article does not cover CNI installation or packet paths. For Pod-to-Pod connectivity concepts and Service routing, read Kubernetes networking.
Discover CNI DaemonSets and node agents
List DaemonSets cluster-wide. Networking implementations usually appear as node-level agents:
kubectl get daemonsets -ASample output:
NAMESPACE NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
calico-system calico-node 2 2 2 2 2 kubernetes.io/os=linux 3d1h
calico-system csi-node-driver 2 2 2 2 2 kubernetes.io/os=linux 3d1h
kube-system kube-proxy 2 2 2 2 2 kubernetes.io/os=linux 3d1hcalico-node is the Calico CNI node agent in this lab. kube-proxy handles Service forwarding and is not a CNI plugin, but it often appears in the same inventory pass. The csi-node-driver DaemonSet is Calico’s ephemeral CSI node plugin — separate from the CNI agent — and is covered in the CSI section.
Inspect CNI configuration on the node
SSH to a worker and list the active CNI definitions:
Cluster node access in this lab assumes SSH from your workstation.
sudo ls -l /etc/cni/net.d/Sample output:
total 8
-rw-------. 1 root root 735 Jul 25 08:45 10-calico.conflist
-rw-------. 1 root root 2801 Jul 27 14:57 calico-kubeconfigThe filename is only a clue. Inspect the file contents to see which plugin chain the runtime loads:
sudo cat /etc/cni/net.d/10-calico.conflistTrimmed sample from this lab:
{
"name": "k8s-pod-network",
"cniVersion": "0.3.1",
"plugins": [
{
"type": "calico",
"ipam": { "type": "calico-ipam", ... },
"kubernetes": {
"kubeconfig": "/etc/cni/net.d/calico-kubeconfig"
},
...
},
{
"type": "portmap",
"capabilities": { "portMappings": true },
"snat": true
}
]
}Inspect the plugins[].type entries to identify the actual chain. In this lab they include calico and portmap. Merely finding binaries under /opt/cni/bin does not mean containerd invokes all of them; only plugins referenced by the loaded configuration participate.
The kubeconfig lets the Calico CNI plugin reach the Kubernetes API for operations such as IP allocation and WorkloadEndpoint registration. The calico-node agent, including Felix, programs routes and enforces network policy on the node.
List plugin binaries:
sudo ls -l /opt/cni/bin/ | head -10Sample output:
total 263612
-rwxr-xr-x. 1 root root 3678488 Apr 24 16:08 bandwidth
-rwxr-xr-x. 1 root root 4169208 Apr 24 16:08 bridge
-rwxr-xr-x. 1 root root 95238822 Jul 24 13:56 calico
-rwxr-xr-x. 1 root root 95238822 Jul 24 13:56 calico-ipam
-rwxr-xr-x. 1 root root 10389208 Apr 24 16:08 dhcp
...Several standard reference plugins (bridge, loopback, host-local, bandwidth) ship alongside the Calico-specific binaries. Presence on disk is not the same as participation in the active conflist.
Match running CNI pods to nodes
Correlate API objects with processes:
kubectl get pods -n calico-system -l k8s-app=calico-node -o wideSample output:
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
calico-node-hc7ph 1/1 Running 1 (2d6h ago) 3d1h 192.168.56.109 worker01 <none> <none>
calico-node-sc7p2 1/1 Running 0 39h 192.168.56.108 k8s-cp <none> <none>One calico-node Pod runs per node, which is the usual DaemonSet pattern for CNI agents.
How CSI integrates storage
CSI is the Container Storage Interface. Kubernetes core objects — PersistentVolumeClaim, PersistentVolume, StorageClass — stay the same for persistent volumes; a vendor or platform driver implements the CSI RPCs. CSI also supports inline ephemeral volumes defined directly in a Pod spec.
A typical CSI deployment includes:
- Controller plugin — provisioning, attachment, and control-plane operations (often a Deployment with sidecars)
- Node plugin — mount and unmount on workers (often a DaemonSet)
- Sidecar containers — connect Kubernetes controllers (external-provisioner, attacher, resizer) to the driver gRPC socket
CSI controller components are optional, and attachment is driver-dependent. Inline ephemeral CSI volumes bypass PVCs, PVs, StorageClasses, and controller provisioning.
Persistent dynamically provisioned CSI volume:
PVC
↓
StorageClass and external-provisioner
↓
PV
↓
CSI controller plugin
↓
VolumeAttachment, when required
↓
kubelet and CSI node plugin
↓
Pod mountInline ephemeral CSI volume:
Pod spec with csi volume
↓
kubelet
↓
CSI node plugin
↓
Pod mountStatically pre-provisioned CSI PersistentVolumes skip dynamic provisioning but still use CSI node operations and, when required, attachment. Non-CSI volumes such as hostPath, local, NFS, Secrets, and ConfigMaps do not enter the CSI path. For hands-on PVC and PV objects without driver installation, see Kubernetes PersistentVolume and PVC.
Discover installed CSI drivers
CSI surfaces through Kubernetes API objects and through Pods in driver namespaces. Work through the read-only resources first.
List CSIDriver and CSINode objects
kubectl get csidriversSample output:
NAME ATTACHREQUIRED PODINFOONMOUNT STORAGECAPACITY TOKENREQUESTS REQUIRESREPUBLISH MODES AGE
csi.tigera.io true true false <unset> false Ephemeral 3d1hThe only registered CSI driver in this lab is csi.tigera.io. Its MODES value is Ephemeral, meaning it handles inline volumes defined directly in Pod specifications rather than PVC-backed persistent disks. Calico uses this driver for node-local Unix socket volumes; it is not the storage backend behind the local-path StorageClass.
List per-node driver registration with driver names, not only a count:
kubectl get csinodes -o custom-columns='NODE:.metadata.name,DRIVERS:.spec.drivers[*].name'Sample output:
NODE DRIVERS
k8s-cp csi.tigera.io
worker01 csi.tigera.ioCSINode.spec.drivers records the registered CSI driver names and node-specific IDs.
Read StorageClass provisioners
kubectl get storageclassesSample output:
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
local-path (default) rancher.io/local-path Delete WaitForFirstConsumer false 37hThe PROVISIONER column is the string Kubernetes matches to a driver or external provisioner. The rancher.io/local-path provisioner dynamically creates hostPath or local PersistentVolumes through the Local Path Provisioner controller. It is not a CSI driver, so it does not register a corresponding CSIDriver object. A CSI-backed StorageClass normally uses the CSI driver name as its provisioner value.
Describe a class when you need binding mode or default-class annotations:
kubectl describe storageclass local-pathThe Provisioner field repeats the same identifier controllers use when creating PersistentVolumes. In this lab it uses a separate Local Path Provisioner Deployment rather than CSI.
Check VolumeAttachment objects
kubectl get volumeattachmentsOn a quiet lab cluster this list may be empty until a volume that requires explicit attachment is in use. When present, each row links a PersistentVolume to a node for drivers that set attachRequired: true. Because csi.tigera.io advertises only the Ephemeral lifecycle mode, it cannot back PVC or PV workflows. Inline ephemeral CSI volumes are published by kubelet through the CSI node service and do not create VolumeAttachment objects. The volumeLifecycleModes field controls whether a driver supports persistent or inline ephemeral volumes, and inline ephemeral volumes are defined directly in the Pod.
Find CSI Pods by namespace and labels
Drivers run as Pods. Search by namespace first:
kubectl get pods -n calico-system | grep csiSample output:
csi-node-driver-54j2p 2/2 Running 0 39h
csi-node-driver-6c9pk 2/2 Running 2 (2d6h ago) 3d1hThe csi-node-driver DaemonSet runs Calico’s ephemeral CSI node plugin and its registrar. It supplies node-local socket volumes for Calico integration; it is separate from the calico-node CNI agent and does not provision the local-path persistent volumes shown in this lab. Other installations use namespaces such as kube-system or a vendor name. Combine kubectl get pods -A with grep csi when you do not know the namespace yet.
| Object | Information it carries |
|---|---|
CSIDriver |
Cluster-wide driver capabilities, modes, and attachment behavior |
CSINode |
Drivers registered on an individual node |
StorageClass |
Provisioner string and volume parameters for dynamic provisioning |
VolumeAttachment |
Requested attachment of a volume to a specific node |
Quick CNI, CSI and CRI inspection checklist
Repeat this compact pass on any cluster built with install Kubernetes with kubeadm. Replace worker01 with your node name. Detailed commands and sample output live in the CRI, CNI, and CSI sections above.
- Read
.status.nodeInfo.containerRuntimeVersion. - Read the kubelet
containerRuntimeEndpoint(APIconfigzor/var/lib/kubelet/instance-config.yaml). - Query the exact endpoint returned by Step 2. For this tested containerd node:
RUNTIME_ENDPOINT=unix:///var/run/containerd/containerd.sock
sudo crictl \
--runtime-endpoint "$RUNTIME_ENDPOINT" \
versionOn a CRI-O node, set RUNTIME_ENDPOINT to the CRI-O socket reported by kubelet instead.
4. Inspect the active .conf or .conflist contents and compare them with the cniconfig section from crictl info. Do not infer the active chain only from filenames or the binaries installed under /opt/cni/bin.
5. Match the CNI plugin chain to its node DaemonSet.
6. List CSIDriver and per-node CSINode.spec.drivers. Record the driver names. A CSI-backed StorageClass normally uses the same driver name in its provisioner field, but non-CSI provisioners such as rancher.io/local-path will not match a CSIDriver.
7. Determine whether each StorageClass is CSI-backed or uses another provisioner (kubectl get storageclasses and, when needed, provisioner Pods such as kubectl get pods -n local-path-storage).
When a PVC stays Pending, compare its storageClassName to that provisioner table and to the running provisioner software. The provisioner string is the link between API objects and the storage path actually in use.
Interface failure boundaries
Symptoms that look similar from kubectl get pods often belong to different interfaces. Use this table to pick a starting direction before you open vendor logs.
| Symptom | Primary direction |
|---|---|
| Node cannot connect to runtime | CRI / container runtime socket and service |
FailedCreatePodSandBox |
Inspect the full event first: CRI sandbox creation failed; investigate CNI only when the message mentions network setup or a CNI plugin |
| PVC remains Pending | StorageClass, provisioner, or capacity |
FailedAttachVolume |
CSI controller or VolumeAttachment |
FailedMount |
CSI node plugin, kubelet mount, or volume configuration |
Node-wide NotReady status often traces to kubelet or CRI before you inspect individual Pods. Sandbox errors that mention network setup, IP allocation, or a named CNI plugin point to the CNI configuration and node agent. Other FailedCreatePodSandBox messages can originate from the CRI runtime itself. Volume events that mention attach or mount reference CSI and the PersistentVolume and PVC object chain — unless the volume is non-CSI, in which case CSI objects will not explain the failure.
For Pod-level scheduling and image pull issues that are not interface-specific, read Pod stuck in ContainerCreating.
What's Next
- Add, Remove, Reset and Rejoin Kubernetes Nodes
- Cordon, Drain and Uncordon Kubernetes Nodes
- Configure a Highly Available Kubernetes Control Plane with kubeadm
References
- Container Runtime Interface (CRI) — kubelet to runtime contract
- Debugging Kubernetes nodes with crictl — configure and query a CRI endpoint
- Kubelet configuration —
containerRuntimeEndpointfield - Kubelet authentication and authorization — configz and node subresource permissions
- Network Plugins (CNI) — Kubernetes CNI requirements
- CNI specification and libraries — CNI runtime and plugin contract
- containerd CRI configuration — CNI directories and configuration loading
- Container Storage Interface (CSI) — CSI driver model and sidecars
- Deploying a CSI driver — controller and node plugin architecture
- CSIDriver API — persistent and ephemeral lifecycle modes
- Calico CSI integration — Calico socket-volume CSI purpose
- Local Path Provisioner — hostPath and local PersistentVolume provisioning
- Node status fields —
containerRuntimeVersionand related node information
Summary
Kubernetes delegates three node-level concerns to pluggable interfaces. CRI connects kubelet to a container runtime such as containerd for image pulls and container lifecycle. CNI configures Pod network interfaces when the runtime creates a sandbox — the loaded conflist under /etc/cni/net.d and node agents such as calico-node implement that layer. CSI links volume operations to drivers through API objects like CSIDriver and CSINode, whether the path is persistent PVC provisioning or an ephemeral inline volume.
You practiced reading those layers from the outside in: Node status and endpoint-qualified crictl for the runtime, DaemonSets and conflist contents for networking, and CSIDriver plus CSINode driver names for storage — without treating Calico’s ephemeral CSI driver as the backend for the local-path StorageClass. Keeping the boundaries straight matters because a CRI outage blocks every Pod on a node, a CNI misconfiguration surfaces as sandbox errors, and CSI or non-CSI provisioner problems appear as Pending claims or mount failures — different log paths.
When you troubleshoot, map the symptom to the interface first, then open the matching discovery commands from this walkthrough. Install and deep configuration guides live elsewhere — build the lab with install Kubernetes with kubeadm, then continue with Kubernetes networking and PersistentVolume and PVC when you need connectivity or storage object detail beyond identification.

