Add, Remove, Reset and Rejoin Kubernetes Nodes

This walkthrough uses an existing kubeadm control plane (k8s-cp) and a prepared worker host (worker01) that is not yet registered. You will join that worker, cordon and drain it, reset the host, delete the Node object, and then rejoin it with a fresh bootstrap token.

Understand the Worker Node Lifecycle

A kubeadm worker moves through distinct phases. Skipping a step—especially drain before reset, or reset before kubectl delete node—leaves Pods stranded or lets a running kubelet re-register a Node object you thought you removed.

Prepare node
  ↓
kubeadm join
  ↓
Node becomes Ready
  ↓
cordon and drain
  ↓
kubeadm reset on the worker
  ↓
delete Node object
  ↓
repair or repurpose host
  ↓
generate fresh join command
  ↓
rejoin and wait for Ready

The lifecycle breaks down into three operations:

• Adding capacity — prepare a new host and run kubeadm join
• Removing capacity — drain workloads, reset the host, then delete the Node API object
• Rejoining — reset plus a new join command, not editing the old Node record

Stacked-etcd control-plane removal changes quorum membership and must follow the HA procedure in high availability with kubeadm. It does not reuse the worker-only drain-reset-delete pattern described below.

Add a Worker Node

Prepare the Host and Check Version Skew

Before kubeadm join, the host must match what the control plane expects. Full node preparation—swap, kernel modules, sysctl, containerd, and package install—is in install Kubernetes with kubeadm. For join-specific checks, confirm the following on the worker candidate:

• The kubeadm binary used for join matches the last kubeadm minor version used to initialize or upgrade the cluster
• The kubelet must not be newer than the API server
• Kubelet 1.25 and later may be up to three minor releases older; earlier kubelet versions may only be up to two minor releases older
• Using the same current patch release for kubeadm and kubelet is the simplest lab configuration
• containerd (or your chosen CRI runtime) is installed, enabled, and exposes the socket kubelet expects
• swap matches the kubelet configuration; with the default failSwapOn behaviour, disable swap before joining
• br_netfilter plus bridge sysctl settings are applied
• the node name or hostname is unique and stable
• the joining host can resolve and reach the API server endpoint and port printed by kubeadm token create --print-join-command
• no stale kubeadm-managed join files remain, especially /etc/kubernetes/kubelet.conf, /etc/kubernetes/bootstrap-kubelet.conf, or /etc/kubernetes/pki

Check versions before join. On the joining worker, confirm the kubeadm binary matches the cluster:

kubeadm version -o short

Sample output:

v1.36.3

On the control plane, confirm the same kubeadm minor version:

sudo kubeadm version -o short

Sample output:

v1.36.3

Confirm the kubelet binary on the worker stays within the supported skew relative to the API server:

kubelet --version

Sample output:

Kubernetes v1.36.3

List current members from your workstation to confirm only the control plane is registered before you join the worker:

kubectl get nodes

Sample output:

NAME     STATUS   ROLES           AGE    VERSION
k8s-cp   Ready    control-plane   3d1h   v1.36.3

In this lab, no newer kubeadm package has been staged on the control plane, so both commands reporting v1.36.3 confirms the join-version match. In another cluster, compare against the kubeadm minor version used for the last successful kubeadm init or kubeadm upgrade, not only whichever kubeadm binary is currently installed.

The Node VERSION column shown by kubectl reports kubelet version and is used only for the kubelet skew check.

Generate and Run the Join Command

Bootstrap tokens expire (default 24 hours). If you lost the original kubeadm init join output, create a new token on a control-plane node:

sudo kubeadm token create --print-join-command

Sample output:

kubeadm join 192.168.56.108:6443 --token 51sy96.r2ay7yupb9ishpo1 --discovery-token-ca-cert-hash sha256:359730c9d6df38487db5499627b06cbbeea3fad7b53df4daa28bd157c878c79b

Copy the entire printed line to the worker; it includes the API endpoint, token, and CA hash.

The printed line contains three discovery pieces:

• API server endpoint (host:6443)
• bootstrap token ID and secret
• discovery-token-ca-cert-hash so the joining node trusts the cluster CA

Run the full command on the worker as root. Generating a new token does not reinitialize the cluster or restart etcd.

On the prepared worker, run the printed join line. It registers the node and starts TLS bootstrap:

sudo kubeadm join 192.168.56.108:6443 --token 51sy96.r2ay7yupb9ishpo1 --discovery-token-ca-cert-hash sha256:359730c9d6df38487db5499627b06cbbeea3fad7b53df4daa28bd157c878c79b

Sample output (trimmed):

[preflight] Running pre-flight checks
[kubelet-start] Starting the kubelet
[kubelet-check] The kubelet is healthy after 1.00923153s
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

The join message confirms bootstrap finished; the node may still show NotReady until the CNI plugin initializes on the node.

Wait for the Node to Become Ready

The node's CNI plugin must initialize before the node normally reaches Ready. In many kubeadm clusters, including this lab, that initialization depends on a node-level CNI DaemonSet starting successfully. Until then you may see NetworkPluginNotReady. Block until the Ready condition is true:

kubectl wait --for=condition=Ready node/worker01 --timeout=180s

Sample output:

node/worker01 condition met

The node passed the Ready check; networking should be usable for new Pods.

Confirm the joined node reports Ready with its kubelet version:

kubectl get node worker01 -o wide

Sample output:

NAME       STATUS   ROLES           AGE   VERSION   INTERNAL-IP      EXTERNAL-IP   OS-IMAGE     ...
worker01   Ready    <none>          36s   v1.36.3   192.168.56.109   <none>        Rocky Linux 10.2 ...

STATUS should show Ready once kubelet and CNI report healthy.

When status stalls, read the Ready condition directly instead of scrolling through describe output:

kubectl get node worker01 -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}{"\n"}'

Sample output:

True

A value of True means the kubelet heartbeats and CNI checks passed.

Inspect Events and other conditions when the value is not True:

kubectl describe node worker01

Look for NetworkUnavailable=False when your networking implementation reports that condition, and inspect CNI-related Events.

Troubleshoot Join Failures

Remove and Reset a Worker Node

Cordon and Drain

Before you reset a host, stop new Pods from landing on it and evict workloads you can move. kubectl drain marks the node unschedulable itself, evicts eligible Pods through the Eviction API, respects PDBs and waits for the original Pods to terminate.

kubectl drain handles some Pod types differently:

• mirror Pods are never deleted because they cannot be removed through the API
• DaemonSet Pods remain; --ignore-daemonsets allows the drain to continue despite them
• --force is used when unmanaged Pods or Pods whose controller no longer exists would otherwise block the drain

Minimum sequence — see cordon, drain and uncordon for flag detail, PDB behavior, and DaemonSet handling. Mark the node unschedulable first:

kubectl cordon worker01

Sample output:

node/worker01 cordoned

New Pods will not schedule on this node until you uncordon it.

Confirm scheduling is disabled:

kubectl get nodes

Sample output:

NAME       STATUS                     ROLES           AGE    VERSION
k8s-cp     Ready                      control-plane   3d1h   v1.36.3
worker01   Ready,SchedulingDisabled   <none>          3d1h   v1.36.3

SchedulingDisabled confirms the cordon took effect.

Evict workloads that can move. DaemonSet Pods remain on the node; --ignore-daemonsets allows the drain to continue despite them. Use --delete-emptydir-data when you accept the loss of emptyDir data:

kubectl drain worker01 --ignore-daemonsets --delete-emptydir-data

Sample output varies with the installed CNI, storage, monitoring and ingress add-ons:

node/worker01 already cordoned
Warning: ignoring DaemonSet-managed Pods: calico-system/calico-node-hc7ph, calico-system/csi-node-driver-6c9pk, kube-system/kube-proxy-m2w8v
evicting pod local-path-storage/local-path-provisioner-5f584dbd7d-6pgv6
evicting pod kube-system/metrics-server-6c9576559c-tpts9
evicting pod ingress-nginx/ingress-nginx-controller-59d9744bfb-7nhdf
pod/metrics-server-6c9576559c-tpts9 evicted
pod/local-path-provisioner-5f584dbd7d-6pgv6 evicted
pod/ingress-nginx-controller-59d9744bfb-7nhdf evicted
node/worker01 drained

In this two-node lab:

• worker01 may be the only schedulable worker
• if the control-plane node retains its default NoSchedule taint, controllers can create replacement Pods but those Pods may remain Pending until another worker joins
• a successful drain confirms that the original Pods were evicted; it does not guarantee that every replacement became Ready elsewhere

Reset the Worker

On the host you drained, run kubeadm reset as root before deleting the Node object. A kubelet with surviving credentials can self-register and recreate the Node API record if you delete the object first:

sudo kubeadm reset -f

Sample output (trimmed):

[reset] Stopping the kubelet service
[reset] Unmounting mounted directories in "/var/lib/kubelet"
[reset] Deleting contents of directories: [/etc/kubernetes/manifests /var/lib/kubelet /etc/kubernetes/pki]
[reset] Deleting files: [/etc/kubernetes/kubelet.conf /etc/kubernetes/bootstrap-kubelet.conf ...]

The reset process does not perform cleanup of CNI plugin configuration,
network filtering rules and kubeconfig files.

Kubelet config and kubeadm-managed PKI paths are removed; CNI and firewall artifacts may remain as the message warns.

Reset is best-effort. It typically removes kubeadm-managed kubelet config and PKI paths but may leave:

• files under /etc/cni/net.d and binaries under /opt/cni/bin
• iptables or nftables rules from CNI or kube-proxy
• administrator kubeconfig files under $HOME/.kube
• application data outside /var/lib/kubelet

For a clean rejoin you usually need reset plus verifying containerd is healthy—not manually deleting every CNI artifact unless join still fails.

Delete the Node Object

After reset completes on the worker, delete the Node API object from the workstation or control plane:

kubectl delete node worker01

Sample output:

node "worker01" deleted

The API no longer lists the worker. kubectl delete waits by default until the object is removed from the API.

kubectl delete node does not clean up the host:

• the VM or bare-metal host keeps running; containerd stays up
• container images remain on disk
• deleting the Node object does not terminate the underlying cloud instance or remove it from an autoscaling group; use the cloud provider, node-group, or autoscaling tooling to deprovision it separately
• managed automation may later replace or reconcile the instance, but kubectl delete node is not an infrastructure-deletion command
• on bare metal or VMs you must clean the OS yourself

Confirm only the remaining members are registered:

kubectl get nodes -o wide

Sample output:

NAME     STATUS   ROLES           AGE    VERSION   INTERNAL-IP      EXTERNAL-IP   OS-IMAGE     ...
k8s-cp   Ready    control-plane   3d1h   v1.36.3   192.168.56.108   <none>        Rocky Linux 10.2 ...

Only the control-plane member should remain after a successful removal.

Rejoin a Worker Node

Verify the Reset State

Treat a rebuilt worker like a new node. Before kubeadm join, verify:

• kubeadm reset completed and /etc/kubernetes/pki on the worker is gone or recreated only by the next join
• the kubelet package and systemd unit are installed
• after reset the kubelet may remain inactive or restart because its kubeadm-managed configuration is absent; kubeadm join writes the new configuration and restarts it
• CNI conflist matches the cluster (often unchanged after reset; reinstall if join errors mention CNI)
• hostname still matches what you intend (worker01 vs FQDN must stay consistent with your install guide)
• no duplicate Node object exists in the API (kubectl get nodes should not list the host)
• system clock is synchronized (bootstrap TLS fails on skewed clocks)

Watch TLS Bootstrap and CSRs

A default kubeadm cluster automatically approves the bootstrap kubelet CSR. If that auto-approval binding was removed, the CSR remains Pending and kubeadm join waits.

Before running join, open another terminal on the workstation and watch CSRs as bootstrap proceeds:

kubectl get csr --watch

Leave that terminal open; you should see a kubelet client CSR move to Approved during join.

Generate a fresh join command on the control plane:

sudo kubeadm token create --print-join-command

Copy the full one-line output and run it on the worker. When auto-approval is disabled, approve the pending CSR manually. List pending requests:

kubectl get csr

Approve the kubelet client CSR by name:

kubectl certificate approve <csr-name>

Replace <csr-name> with the Pending CSR from the watch output; kubeadm join should finish once the certificate is issued.

Sample output after rejoin (auto-approval path):

NAME        AGE   SIGNERNAME                                    REQUESTOR                 REQUESTEDDURATION   CONDITION
csr-gpq2b   38s   kubernetes.io/kube-apiserver-client-kubelet   system:bootstrap:51sy96   <none>              Approved,Issued

On the node after rejoin:

• new kubelet client certificates appear under /var/lib/kubelet/pki/ with a current timestamp
• the Node object's CreationTimestamp resets because registration is new, even when the hostname matches the old member

Wait for Ready and Test Scheduling

Wait for the node to report Ready before you schedule a verification Pod:

kubectl wait --for=condition=Ready node/worker01 --timeout=180s

Sample output:

node/worker01 condition met

The rejoined node is schedulable again.

Read the Ready condition without parsing describe output:

kubectl get node worker01 -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}{"\n"}'

Sample output:

True

True confirms kubelet registration and CNI health after rejoin.

Pin a test Pod to the rejoined node. Read the hostname label first:

NODE_HOSTNAME=$(kubectl get node worker01 -o jsonpath='{.metadata.labels.kubernetes\.io/hostname}')

The label value is what nodeSelector must match; it can differ from the Node object name when FQDN registration is enabled.

Schedule the verification Pod with an explicit command:

kubectl run lifecycle-test --image=busybox:1.36.1 --restart=Never --overrides="{\"spec\":{\"nodeSelector\":{\"kubernetes.io/hostname\":\"${NODE_HOSTNAME}\"}}}" --command -- sleep 3600

Sample output:

pod/lifecycle-test created

Block until the Pod reports Ready:

kubectl wait --for=condition=Ready pod/lifecycle-test --timeout=120s

Sample output:

pod/lifecycle-test condition met

Confirm the Pod landed on the intended node:

kubectl get pod lifecycle-test -o wide

A healthy result shows:

• NODE set to worker01
• STATUS of Running with a Pod IP assigned

For a CNI-neutral check of system Pods on the node:

kubectl get pods -A -o wide --field-selector spec.nodeName=worker01

You should see the node-level agents expected by your cluster, such as CNI DaemonSet Pods and kube-proxy when kube-proxy is deployed rather than replaced by another Service dataplane.

Remove the verification Pod when you are done:

kubectl delete pod lifecycle-test

The verification Pod is removed; the rejoined node stays registered in the cluster.

Decide Whether to Repair or Rejoin

When kubelet cannot authenticate to the API server, the node flaps NotReady or stays absent from kubectl get nodes. Work through these checks before editing files by hand:

1. Confirm time sync on the node (timedatectl status).
2. Inspect kubelet logs: journalctl -u kubelet -e.
3. List CSRs: kubectl get csr — pending kubelet client CSRs may need approval on clusters without auto-approval.
4. Inspect the rotating kubelet client certificate directly:

sudo openssl x509 -in /var/lib/kubelet/pki/kubelet-client-current.pem -noout -subject -issuer -dates

Check that notAfter is in the future and the subject matches the node identity you expect.

5. If bootstrap kubeconfig is missing and multiple cert paths are inconsistent, kubeadm reset followed by a new join is more reliable than copying individual PEM files from another node.

Deep PKI paths, kubeadm certs renew, and kubeconfig users are out of scope here—use a PKI-focused lesson when you need file-level renewal without removing the node.

Control-Plane Node Boundary

Worker nodes and control-plane nodes follow different join and removal rules:

• append --control-plane to kubeadm join
• upload or copy shared certificates from the first control-plane (--control-plane --certificate-key flow for stacked etcd)
• point join at the HA API endpoint (load balancer VIP), not only the first master IP
• stacked etcd adds a member to the quorum; external etcd uses a different topology

HA and certificate limits to keep in mind:

• a cluster initialized without --control-plane-endpoint cannot be converted into HA merely by joining another node
• kubeadm explicitly does not support converting a single-control-plane cluster created without a shared control-plane endpoint into HA
• certificates uploaded to the encrypted kubeadm-certs Secret are deleted after two hours; the printed certificate key is usable only while that Secret exists
• run kubeadm init phase upload-certs --upload-certs to upload certificates again and generate a new key
• worker lifecycle in this article—drain, kubeadm reset, kubectl delete node, rejoin—does not replace an HA bootstrap procedure
• stacked-etcd control-plane removal changes quorum membership and must follow the HA procedure in high availability with kubeadm

Troubleshooting Boundaries

What's Next

• Cordon, Drain and Uncordon Kubernetes Nodes
• Configure a Highly Available Kubernetes Control Plane with kubeadm
• Back Up and Restore Kubernetes etcd

References

• Adding nodes with kubeadm — official join workflow and CSR auto-approval
• kubeadm reset — cluster cleanup on a removed node
• Version skew policy — supported kubeadm and kubelet versions for join
• Certificate management with kubeadm — control-plane certificate upload and expiry
• Creating highly available clusters with kubeadm — HA endpoint and stacked-etcd requirements
• Kubelet certificate rotation — rotating kubelet client certificates
• Safely drain a node — upstream drain semantics
• kubectl wait — wait for Ready and delete conditions
• Node management — Node object lifecycle

Summary

You walked the full kubeadm worker lifecycle on a live cluster: prepare the host using the same baseline as install Kubernetes with kubeadm, confirm kubeadm and kubelet versions against the skew policy, generate a join command with kubeadm token create --print-join-command, and register the node until kubelet completes TLS bootstrap and CNI reports healthy. Watch CSRs in a second terminal when you want to see bootstrap approval in real time.

Removing a node means cordon and drain first so controllers reschedule workloads, then kubeadm reset on the host so kubelet cannot self-register, then kubectl delete node to drop the API registration. kubeadm reset clears kubeadm-managed kubelet state but not every CNI file or firewall rule. Rejoining is reset plus a fresh token, not resurrecting the old Node UID.

After rejoin, wait for Ready with kubectl wait, confirm kubelet client certificates under /var/lib/kubelet/pki/, inspect system Pods on the node, and pin a test Pod before you declare the lifecycle complete. Control-plane expansion, HA conversion without a shared endpoint, and certificate surgery sit outside this worker-focused path. When skew or version checks matter before you add nodes, read check Kubernetes cluster version and plan upgrades with Kubernetes upgrade with kubeadm before you join mismatched kubelet packages.