Kubernetes Node NotReady [kubelet, Runtime and Pressure Fixes]

Tested on	Rocky Linux 10.2 (Red Quartz) workstation
Package	`kubectl 1.36.3` `kubelet 1.36.3` `containerd 2.2.5`
Applies to	Ubuntu, Debian, Kali Linux, Linux Mint, Pop!_OS, Raspberry Pi OS, elementary OS, Zorin OS, Parrot OS, MX Linux, RHEL, Rocky Linux, AlmaLinux, Oracle Linux, CentOS Stream, Fedora Kubernetes cluster
Cert prep	CKA
Lab environment	Multi-node kubeadm cluster with containerd — install Kubernetes with kubeadm. SSH access to worker nodes is used for kubelet and runtime checks; `kubectl debug node` is shown as the no-SSH path.
Privilege	Normal user for `kubectl`; sudo or root on nodes for `systemctl`, `journalctl`, and `crictl`
Scope	Worker Node Ready, NotReady, and Unknown status, Node conditions and automatic taints, kube-node-lease heartbeats, kubelet and containerd health, CRI socket checks, memory disk and PID pressure, API connectivity from the node, kubelet certificates and clock skew, kubectl debug node with /host and sysadmin profile, cordon or drain decisions, and recovery verification. Does not cover static control-plane Pods, API server etcd or scheduler repair, full CNI troubleshooting, OS install, hardware replacement, or cluster autoscaler.
Related guides	kubectl debug Pods

A worker that flips to NotReady stops accepting new work and often evicts running Pods. Start from the Node object and its Lease, then move onto the host—or a node-debug Pod—to check kubelet, the container runtime, pressure, and API connectivity.

IMPORTANT

This article covers worker node health: kubelet heartbeats, runtime, pressure, and node certificates. It does not repair static control-plane Pods, etcd, or the API server process. Those belong in control plane troubleshooting.

Ready, NotReady, and Unknown

The Ready condition is the main signal, but kubectl get nodes STATUS is a summary of several conditions:

Condition	Meaning
`Ready=True`	kubelet is posting healthy status
`Ready=False`	kubelet is posting and reports the node unhealthy
`Ready=Unknown`	control plane stopped receiving status updates

On this lab, stopping the worker kubelet produced Ready=Unknown with reason NodeStatusUnknown, while kubectl get nodes still printed NotReady. Treat the condition fields as the source of truth, not the single STATUS word alone.

Inspect the Node object

Describe the node and read Conditions, Taints, Addresses, Capacity, Allocatable, and System Info:

bash
kubectl describe node worker01 | sed -n '1,75p'

output

Name:               worker01
Roles:              <none>
...
Taints:             <none>
Unschedulable:      false
Lease:
  HolderIdentity:  worker01
  AcquireTime:     <unset>
  RenewTime:       Mon, 27 Jul 2026 20:18:09 +0530
Conditions:
  Type                 Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----                 ------  -----------------                 ------------------                ------                       -------
  NetworkUnavailable   False   Mon, 27 Jul 2026 18:23:20 +0530   Mon, 27 Jul 2026 18:23:20 +0530   CalicoIsUp                   Calico is running on this node
  MemoryPressure       False   Mon, 27 Jul 2026 20:13:46 +0530   Mon, 27 Jul 2026 18:11:50 +0530   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure         False   Mon, 27 Jul 2026 20:13:46 +0530   Mon, 27 Jul 2026 18:11:50 +0530   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure          False   Mon, 27 Jul 2026 20:13:46 +0530   Mon, 27 Jul 2026 18:11:50 +0530   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready                True    Mon, 27 Jul 2026 20:13:46 +0530   Mon, 27 Jul 2026 20:07:38 +0530   KubeletReady                 kubelet is posting ready status
Addresses:
  InternalIP:  192.168.56.109
  Hostname:    worker01
Capacity:
  cpu:                1
  ephemeral-storage:  14343176Ki
  memory:             2008296Ki
  pods:               110
System Info:
  Container Runtime Version:  containerd://2.2.5
  Kubelet Version:            v1.36.3

Healthy output shows Ready=True, no unexpected taints, a recent Lease renew time, and a known container runtime version. Pull the same conditions in a compact form when you need a quick paste for a ticket:

bash
kubectl get node worker01 -o jsonpath='{range .status.conditions[*]}{.type}={.status} reason={.reason}{"\n"}{end}'

output

NetworkUnavailable=False reason=CalicoIsUp
MemoryPressure=False reason=KubeletHasSufficientMemory
DiskPressure=False reason=KubeletHasNoDiskPressure
PIDPressure=False reason=KubeletHasSufficientPID
Ready=True reason=KubeletReady

Inspect node heartbeat Leases

kubelet renews a lightweight Lease in kube-node-lease. Stale renewTime is often the earliest API-visible sign that heartbeats stopped.

bash
kubectl get lease -n kube-node-lease worker01 -o yaml | sed -n '1,25p'

output

apiVersion: coordination.k8s.io/v1
kind: Lease
metadata:
  creationTimestamp: "2026-07-27T12:41:51Z"
  name: worker01
  namespace: kube-node-lease
  ownerReferences:
  - apiVersion: v1
    kind: Node
    name: worker01
    ...
spec:
  holderIdentity: worker01
  leaseDurationSeconds: 40
  renewTime: "2026-07-27T14:50:50.272635Z"

Compare renewTime with current UTC time. With leaseDurationSeconds: 40, a renew older than roughly that window means the control plane will mark the node unreachable.

After stopping kubelet on worker01 for this lab, the node looked like this:

bash

kubectl get nodes

output

NAME       STATUS     ROLES           AGE    VERSION
k8s-cp     Ready      control-plane   129m   v1.36.3
worker01   NotReady   <none>          127m   v1.36.3

bash
kubectl get node worker01 -o jsonpath='{.status.conditions[?(@.type=="Ready")].status} {.status.conditions[?(@.type=="Ready")].reason} {.status.conditions[?(@.type=="Ready")].message}{"\n"}'

output

Unknown NodeStatusUnknown Kubelet stopped posting node status.

The Lease renew stopped advancing while cluster time moved on—exactly the heartbeat gap you are hunting.

Understand automatic Node taints

When conditions degrade, the node controller applies taints that change scheduling and eviction:

Taint key	Typical trigger
`node.kubernetes.io/not-ready`	Ready is False
`node.kubernetes.io/unreachable`	Ready is Unknown / lost heartbeat
`node.kubernetes.io/memory-pressure`	MemoryPressure is True
`node.kubernetes.io/disk-pressure`	DiskPressure is True
`node.kubernetes.io/pid-pressure`	PIDPressure is True

Effects matter:

NoSchedule — new Pods without a matching toleration stay off the node
NoExecute — existing Pods without tolerations can be evicted after a grace period

With kubelet stopped, this lab applied both effects for unreachable:

output

Taints:             node.kubernetes.io/unreachable:NoExecute
                    node.kubernetes.io/unreachable:NoSchedule

Do not remove these taints by hand as a “fix.” Restore kubelet health and let the controller clear them.

Check kubelet service and logs

On the affected node, confirm kubelet is running:

bash

systemctl status kubelet --no-pager | head -12

output

● kubelet.service - kubelet: The Kubernetes Node Agent
     Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; preset: disabled)
    Drop-In: /usr/lib/systemd/system/kubelet.service.d
             └─10-kubeadm.conf
     Active: active (running) since Mon 2026-07-27 20:19:26 IST; 1min 46s ago
...
   Main PID: 742848 (kubelet)

Active: active (running) is required before you chase deeper config issues. Read recent logs for API, certificate, runtime, and registration errors:

bash
journalctl -u kubelet --since "10 min ago" --no-pager | grep -iE 'error|fail|certificate|runtime|cni|register' | tail -20

Look for configuration errors, API connectivity failures, certificate problems, CRI connection errors, cgroup issues, disk or image filesystem problems, and node registration failures. On this lab, stopping and starting kubelet left clear systemd markers:

output

Jul 27 20:18:38 worker01 systemd[1]: Stopped kubelet.service - kubelet: The Kubernetes Node Agent.
Jul 27 20:19:28 worker01 kubelet[742848]: I0727 20:19:28.690771  742848 kubelet.go:2622] "Starting kubelet main sync loop"

Check the container runtime and CRI socket

kubelet cannot run Pods if its configured CRI endpoint is down. Confirm containerd (or your runtime) and crictl:

bash

systemctl status containerd --no-pager | head -12

output

● containerd.service - containerd container runtime
     Loaded: loaded (/usr/lib/systemd/system/containerd.service; enabled; preset: disabled)
     Active: active (running) since Mon 2026-07-27 18:07:48 IST; 2h 10min ago
...
   Main PID: 698994 (containerd)

Inspect the crictl endpoints this node uses:

bash

cat /etc/crictl.yaml

output

runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: unix:///run/containerd/containerd.sock

List running containers and sandboxes:

bash

crictl ps | head -8

output

CONTAINER           IMAGE               CREATED             STATE               NAME                        ATTEMPT             POD ID              POD
8a52731783e3b       38667dd9be96c       7 minutes ago       Running             coredns                     0                   a2e97aca38c70       coredns-5c45547cc5-8t8vq
...
052be76bbf405       c281bc67214e2       2 hours ago         Running             calico-node                 0                   03409f0b4ebec       calico-node-j9m4s

bash

crictl pods | head -8

output

POD ID              CREATED             STATE               NAME                             NAMESPACE              ATTEMPT             RUNTIME
a2e97aca38c70       7 minutes ago       Ready               coredns-5c45547cc5-8t8vq         kube-system            0                   (default)
...
03409f0b4ebec       2 hours ago         Ready               calico-node-j9m4s                calico-system          0                   (default)

If crictl cannot talk to the socket, fix the runtime before blaming Kubernetes scheduling. Align the kubelet --container-runtime-endpoint (or config equivalent) with the socket path in /etc/crictl.yaml.

Check node resource pressure

Memory pressure

Inspect free memory and OOM evidence on the node:

bash

free -h

output

total        used        free      shared  buff/cache   available
Mem:           1.9Gi       1.1Gi       118Mi        39Mi       930Mi       815Mi
Swap:             0B          0B          0B

When MemoryPressure=True, free space for kubelet eviction thresholds, stop runaway workloads, and avoid adding more Guaranteed Pods until available memory recovers.

Disk pressure

Check the root filesystem, image store, and inodes:

bash

df -h / /var/lib/containerd

output

Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/rlm-root   14G  9.2G  3.9G  71% /
/dev/mapper/rlm-root   14G  9.2G  3.9G  71% /

bash

df -i /

output

Filesystem           Inodes  IUsed  IFree IUse% Mounted on
/dev/mapper/rlm-root 917504 152604 764900   17% /

Clean unused images and large container logs when disk or inode pressure is True. Do not ignore a full /var/lib/containerd even if / still has space on nodes that split those mounts.

PID pressure

High process counts from fork bombs or stuck workloads trip PIDPressure. Count processes and stop the offender; kubelet cannot schedule useful work on a PID-exhausted node.

Check network and API connectivity

From the node, confirm the kubelet can reach the API server listed in its kubeconfig:

bash

grep server: /etc/kubernetes/kubelet.conf

output

server: https://192.168.56.108:6443

bash
curl -sk --connect-timeout 3 -o /dev/null -w '%{http_code}\n' https://192.168.56.108:6443/healthz

output

Also verify DNS for the API hostname when you use a name instead of an IP, TCP 6443 through firewalls, node routes, and any HTTP proxy settings. If every node fails the same API check, stop treating this as a single worker fault and open control plane troubleshooting.

Check kubelet certificates and time

Inspect kubelet client material and clock skew—expired or mistimed certs break heartbeats without a “dead” systemd unit.

bash

ls -la /var/lib/kubelet/pki/

output

total 20
drwxr-xr-x.  2 root root 4096 Jul 27 18:11 .
...
-rw-------.  1 root root 1110 Jul 27 18:11 kubelet-client-2026-07-27-18-11-50.pem
lrwxrwxrwx.  1 root root   59 Jul 27 18:11 kubelet-client-current.pem -> /var/lib/kubelet/pki/kubelet-client-2026-07-27-18-11-50.pem
-rw-r--r--.  1 root root 2270 Jul 27 18:11 kubelet.crt

bash

openssl x509 -in /var/lib/kubelet/pki/kubelet-client-current.pem -noout -subject -dates

output

subject=O=system:nodes, CN=system:node:worker01
notBefore=Jul 27 12:36:50 2026 GMT
notAfter=Jul 27 12:36:50 2027 GMT

Confirm the CN matches the node name, the certificate is inside its validity window, and timedatectl shows a sensible clock. Pending CSRs and deeper PKI renewal belong in Kubernetes PKI and certificates.

Debug a node without SSH

When you cannot SSH, create a node debug Pod. Prefer --profile=sysadmin when you need chroot /host:

bash
kubectl debug node/worker01 --image=busybox:1.36 --profile=sysadmin -- sleep 180

output

Creating debugging pod node-debugger-worker01-dhr4b with container debugger on node worker01.

Wait until it is Running, then inspect the host filesystem mounted at /host:

bash

kubectl exec node-debugger-worker01-dhr4b -- ls /host/var/lib/kubelet/pki

output

kubelet-client-2026-07-27-18-11-50.pem
kubelet-client-current.pem
kubelet.crt
kubelet.key

bash

kubectl exec node-debugger-worker01-dhr4b -- chroot /host systemctl is-active kubelet

output

active

Without sysadmin, chroot /host often fails even though /host is visible. Keep the debug image current, delete the debugger Pod when finished, and remember kubelet must be reachable on the node port for kubectl exec to work—if kubelet is down, you still need out-of-band host access.

Decide whether to cordon or drain

Situation	Action
Node briefly degraded but still reachable	Cordon while you investigate
Planned reboot or package update	Drain first when eviction can succeed
Node unreachable / kubelet down	Fix the host; drain may hang
Stateful volumes attached	Confirm application and volume safety before force actions

Maintenance command details live in cordon, drain, and uncordon. Cordon alone does not fix NotReady—it only stops new Pods while you repair the node.

Verify node recovery

After kubelet and the runtime are healthy again, confirm:

bash

kubectl get nodes

output

NAME       STATUS   ROLES           AGE    VERSION
k8s-cp     Ready    control-plane   129m   v1.36.3
worker01   Ready    <none>          128m   v1.36.3

Also confirm the Lease renews, pressure conditions are False, automatic unreachable or not-ready taints are gone, CNI agents are Ready on the node, and a test Pod schedules:

bash
kubectl run node-ok --image=registry.k8s.io/pause:3.10 --restart=Never --overrides='{"spec":{"nodeName":"worker01"}}'

output

pod/node-ok created

bash

kubectl get pod node-ok -o wide

output

NAME      READY   STATUS    RESTARTS   AGE   IP             NODE       NOMINATED NODE   READINESS GATES
node-ok   1/1     Running   0          6s    192.168.5.32   worker01   <none>           <none>

Delete the test Pod when finished. If the node is Ready but new Pods fail sandbox creation, continue with FailedCreatePodSandBox.

References

Summary

Worker NotReady work starts in the API: Ready condition, automatic taints, and the kube-node-lease renew time. Ready=Unknown means heartbeats stopped; Ready=False means the kubelet is still talking and naming a local problem such as pressure or runtime failure.

On the host—or through kubectl debug node with --profile=sysadmin—verify kubelet, containerd, the CRI socket, disk and memory headroom, API reachability on port 6443, and certificate dates. Restoring the service usually clears unreachable taints without manual taint edits.

When the node is Ready again, prove it with a scheduled test Pod and healthy CNI agents. If kubectl cannot reach the API at all, switch to control-plane troubleshooting instead of repeating worker runbooks.

Frequently Asked Questions

1. Why does kubectl get nodes show NotReady when Ready is Unknown?

The STATUS column is a summary. When the control plane stops receiving kubelet status updates, the Ready condition becomes Unknown and kubectl commonly displays NotReady. That is different from Ready=False, where the kubelet is still posting and reporting an unhealthy reason.

2. What is the kube-node-lease used for?

Each node has a Lease in kube-node-lease that the kubelet renews as a lightweight heartbeat. If renewTime falls far behind cluster time, the control plane treats the node as having lost contact even before you dig into full Node status payloads.

3. Should I drain a node that is unreachable?

Drain may not complete normally when the kubelet cannot evict Pods. Controllers may replace workloads because of unreachable NoExecute taints. Prefer fixing kubelet connectivity first, and use cordon or drain only when the node is still reachable enough for eviction to work.

4. When do I need kubectl debug --profile=sysadmin?

A default node debug Pod mounts the host root at /host but is not privileged enough for chroot in many cases. Use the sysadmin profile when you need privileged host inspection such as chroot /host and systemd checks, and only when your cluster policy allows it.

5. Is a NotReady worker the same as a broken API server?

No. Worker NotReady is about that node kubelet, runtime, pressure, or connectivity to the API. If no kubectl command works against the API server, troubleshoot the control plane instead.

Ready, NotReady, and Unknown

Inspect the Node object

Inspect node heartbeat Leases

Understand automatic Node taints

Check kubelet service and logs

Check the container runtime and CRI socket

Check node resource pressure

Memory pressure

Disk pressure

PID pressure

Check network and API connectivity

Check kubelet certificates and time

Debug a node without SSH

Decide whether to cordon or drain

Verify node recovery

References

Summary

Frequently Asked Questions

1. Why does kubectl get nodes show NotReady when Ready is Unknown?

2. What is the kube-node-lease used for?

3. Should I drain a node that is unreachable?

4. When do I need kubectl debug --profile=sysadmin?

5. Is a NotReady worker the same as a broken API server?

Related Articles

Kubernetes Incident Response and Forensics

Kubernetes Runtime Security with Falco

Kubernetes Audit Logging with Policy Examples

Search GoLinuxCloud