Troubleshoot Kubernetes Control Plane Components

When the control plane is sick, decide first whether kubectl can still talk to the API. If it can, use Pods, logs, and /readyz. If it cannot, stay on the control-plane host with kubelet, manifests, and crictl.

Identify the control-plane layout

On a kubeadm control-plane node, kubelet watches static manifests:

ls -la /etc/kubernetes/manifests/

total 24
drwxr-xr-x. 2 root root 4096 Jul 27 20:20 .
drwxr-xr-x. 5 root root 4096 Jul 23 00:02 ..
-rw-------. 1 root root 2604 Jul 27 18:13 etcd.yaml
-rw-------. 1 root root 3877 Jul 27 18:13 kube-apiserver.yaml
-rw-------. 1 root root 3143 Jul 27 18:13 kube-controller-manager.yaml
-rw-------. 1 root root 1929 Jul 27 18:13 kube-scheduler.yaml

/etc/kubernetes/manifests/
├── etcd.yaml
├── kube-apiserver.yaml
├── kube-controller-manager.yaml
└── kube-scheduler.yaml

kubelet creates those Pods. Mirror objects appear in kube-system as names such as kube-apiserver-k8s-cp. Editing a live manifest causes kubelet to recreate the component. Managed Kubernetes control planes usually do not expose this directory.

Decide whether kubectl still works

Branch before you burn time on the wrong tool:

Can kubectl reach the API server?
├── Yes → inspect Pods, Events, logs, /livez and /readyz
└── No  → inspect kubeconfig target, port 6443, kubelet, manifests, and crictl locally

Do not loop on kubectl when the API itself is down. Client kubeconfig mistakes and a dead apiserver look similar from a laptop—verify the server URL and TCP 6443 from a machine that should reach the control plane.

Check API endpoint and load balancer

Confirm what your kubeconfig targets and whether anything answers on 6443. On this single control-plane lab the server is the node IP:

kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}{"\n"}'

https://192.168.56.108:6443

In HA clusters, also verify the load-balancer VIP, backend targets, and certificate SANs covering that name. Differentiate “wrong kubeconfig” from “no apiserver process” before rewriting manifests.

Inspect static Pods through the API

When kubectl works, list control-plane Pods:

kubectl get pods -n kube-system -o wide | grep -E 'NAME|etcd|apiserver|scheduler|controller-manager'

NAME                              READY   STATUS    RESTARTS   AGE     IP               NODE       NOMINATED NODE   READINESS GATES
etcd-k8s-cp                       1/1     Running   0          120m    192.168.56.108   k8s-cp     <none>           <none>
kube-apiserver-k8s-cp             1/1     Running   0          120m    192.168.56.108   k8s-cp     <none>           <none>
kube-controller-manager-k8s-cp    1/1     Running   0          120m    192.168.56.108   k8s-cp     <none>           <none>
kube-scheduler-k8s-cp             1/1     Running   0          94s     192.168.56.108   k8s-cp     <none>           <none>

Describe and read logs for any component that is not Ready. In HA clusters, inspect every replica—not only the first name that appears.

Inspect components locally with crictl

When the API is unavailable—or you want the runtime view—list sandboxes on the control-plane node:

crictl pods | grep -E 'POD ID|etcd|apiserver|scheduler|controller-manager'

POD ID              CREATED              STATE               NAME                                       NAMESPACE            ATTEMPT             RUNTIME
1453b616cce33       16 seconds ago       Ready               kube-scheduler-k8s-cp                      kube-system          2                   (default)
4271c9d8207f9       2 hours ago          Ready               etcd-k8s-cp                                kube-system          0                   (default)
a5c0fd46a3723       2 hours ago          Ready               kube-controller-manager-k8s-cp             kube-system          0                   (default)
0ab281b417ff5       2 hours ago          Ready               kube-apiserver-k8s-cp                      kube-system          0                   (default)

Identify NotReady or missing sandboxes, then use crictl ps and crictl logs <container-id> for the exited container. This path still works when kubectl cannot authenticate to a broken apiserver.

Check kubelet before blaming every component

Static Pods exist only while kubelet can read manifests and talk to the runtime:

systemctl status kubelet --no-pager | head -10

If kubelet is stopped, every control-plane static Pod disappears even when the YAML files are perfect. Treat kubelet here as the static-Pod supervisor. Broader worker recovery remains in Node NotReady.

Validate static Pod manifests

Before rewriting flags, check for:

• YAML syntax errors
• Wrong command-line flags
• Missing hostPath files
• Incorrect certificate paths under /etc/kubernetes/pki
• Invalid volume mounts
• Wrong --advertise-address or etcd endpoints
• Image pull failures
• Duplicate backup files left in the manifests directory

kubelet scans eligible files in that directory. Do not leave kube-scheduler.yaml.bak beside the real manifest—both can be loaded.

Troubleshoot kube-apiserver

When the API answers, start with liveness and readiness:

kubectl get --raw='/livez?verbose' | head -12

[+]ping ok
[+]log ok
[+]loopback-serving-certificate ok
[+]etcd ok
[+]poststarthook/start-apiserver-admission-initializer ok
[+]poststarthook/generic-apiserver-start-informers ok
...

kubectl get --raw='/readyz?verbose' | head -12

[+]ping ok
[+]log ok
[+]etcd ok
[+]etcd-readiness ok
[+]informer-sync ok
...

A failed etcd or etcd-readiness check routes to etcd, not to client networking. Also review serving certificates and SANs, client CA, etcd client certs in the apiserver manifest, port conflicts on 6443, advertise address, and admission or audit config file paths. Read container logs with kubectl when possible, or crictl logs when not.

Troubleshoot etcd

Check the etcd static Pod, data directory, certificates, peer and client URLs, disk space, and ownership. On the control-plane host, use certificate-aware etcdctl:

etcdctl --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  endpoint health

https://127.0.0.1:2379 is healthy: successfully committed proposal: took = 31.335322ms

etcdctl --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  endpoint status --write-out=table

+------------------------+------------------+---------+-----------------+---------+--------+-----------------------+--------+-----------+------------+-----------+------------+--------------------+--------+
|        ENDPOINT        |        ID        | VERSION | STORAGE VERSION | DB SIZE | IN USE | PERCENTAGE NOT IN USE | QUOTA  | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+------------------------+------------------+---------+-----------------+---------+--------+-----------------------+--------+-----------+------------+-----------+------------+--------------------+--------+
| https://127.0.0.1:2379 | 73f02e3a4230ca03 |   3.6.8 |           3.6.0 |   25 MB | 7.1 MB |                   72% | 2.1 GB |      true |      false |         4 |      29243 |              29243 |        |
+------------------------+------------------+---------+-----------------+---------+--------+-----------------------+--------+-----------+------------+-----------+------------+--------------------+--------+

Healthy single-node status shows a leader with no ERRORS column entries. Quorum loss in multi-member etcd needs member-list and peer checks. Snapshot restore procedures stay in etcd backup and restore.

Troubleshoot kube-scheduler

Symptoms include Pending Pods while the API and controllers still look healthy. Confirm the scheduler Pod and its logs first:

kubectl -n kube-system logs kube-scheduler-k8s-cp --tail=8

I0727 14:50:49.049956       1 server.go:178] "Starting Kubernetes Scheduler" version="v1.36.3"
...
I0727 14:50:49.332558       1 leaderelection.go:258] "Attempting to acquire leader lease..." lock="kube-system/kube-scheduler"
I0727 14:50:49.378422       1 leaderelection.go:272] "Successfully acquired lease" lock="kube-system/kube-scheduler"

Also verify the scheduler kubeconfig inside the manifest, API connectivity, leader-election Lease, and any custom scheduler config. Ordinary Pending causes—requests, taints, affinity, unbound PVCs—belong in Pending and ContainerCreating and node affinity, not a scheduler reinstall.

Troubleshoot kube-controller-manager

Controller-manager problems show up as missing reconciliation rather than a single Pod STATUS:

• Deployment replicas not converging
• Node condition or taint actions delayed
• EndpointSlices not updating
• ServiceAccount or certificate controllers idle

Check the static Pod, logs, kubeconfig, cluster-signing certificate paths, leader Lease, and API connectivity. Do not duplicate Deployment-level troubleshooting here—confirm the controller process is leading and free of crash loops first.

A healthy control plane should create ReplicaSets promptly:

kubectl create deploy cp-reconcile --image=registry.k8s.io/pause:3.10 --replicas=2

deployment.apps/cp-reconcile created

kubectl get deploy cp-reconcile

NAME           READY   UP-TO-DATE   AVAILABLE   AGE
cp-reconcile   2/2     2            2           6s

Delete the test Deployment when finished.

Inspect leader election in HA clusters

Scheduler and controller-manager use Lease objects in kube-system:

kubectl get lease -n kube-system | grep -Ei 'scheduler|controller'

kube-controller-manager                k8s-cp_3995ffd1-520c-4ea3-8cd4-3df2907eff6d                                 128m
kube-scheduler                         k8s-cp_48e4928d-e7da-4029-954c-3ae581b13f8b                                 128m

Only one replica holds the lock while others standby. Stale renew times together with a CrashLooping leader Pod explain “API is up but nothing reconciles.” In single-node labs you still see one holder Identity string after each restart.

Controlled failure lab

Use a reversible non-API component. This lab temporarily moved the scheduler manifest:

mv /etc/kubernetes/manifests/kube-scheduler.yaml /root/cp-manifest-backup/

Within a few seconds the mirror Pod disappeared from the API while etcd, apiserver, and controller-manager stayed Running:

try 2: NONE
DURING:
etcd-k8s-cp                       1/1     Running
kube-apiserver-k8s-cp             1/1     Running
kube-controller-manager-k8s-cp    1/1     Running

Locally, crictl still showed a NotReady leftover sandbox until cleanup. Restore immediately:

mv /root/cp-manifest-backup/kube-scheduler.yaml /etc/kubernetes/manifests/

restore 2: kube-scheduler-k8s-cp             0/1   Running   0     4s

Then confirm a new Pod can bind:

kubectl run sched-test --image=registry.k8s.io/pause:3.10 --restart=Never

pod/sched-test created

kubectl get pod sched-test -o wide

NAME         READY   STATUS    RESTARTS   AGE   IP             NODE       NOMINATED NODE   READINESS GATES
sched-test   1/1     Running   0          8s    192.168.5.21   worker01   <none>           <none>

For intentional apiserver or etcd failures, use a disposable lab and keep the restore commands in the same section—do not leave those manifests missing.

Verify full control-plane recovery

Confirm this checklist after repairs:

kubectl get --raw='/readyz' ; echo

Expect ok. Re-run etcd endpoint health, ensure all four manifests exist, and list static Pods as Running. Create a Deployment to prove the controller-manager, schedule a Pod to prove the scheduler, and confirm nodes remain Ready. CoreDNS and kube-proxy should stay Running; if only workers are NotReady, return to the node article.

References

• kubeadm troubleshooting
• Static Pods
• API server health checks
• etcd operations guide
• kube-scheduler
• kube-controller-manager

Summary

Control-plane recovery starts with a reachability branch: use kubectl and /readyz when the API answers, and use kubelet plus crictl on the host when it does not. kubeadm components are static Pods—manifest mistakes and a stopped kubelet take them down together.

etcd health belongs in certificate-aware etcdctl checks and readiness probes on the apiserver. Scheduler and controller-manager failures are confirmed with Pods, logs, and leader Leases before you blame ordinary Pending or Deployment YAML.

A reversible scheduler manifest move is enough to practice static-Pod recovery. After restore, prove /readyz, etcd health, scheduling, and Deployment reconciliation before you call the control plane healthy.