Cordon, Drain and Uncordon Kubernetes Nodes

When you patch a worker, replace a disk, or test kernel settings, you need the node to stop accepting new Pods and shed existing workloads without surprising outages. kubectl cordon, kubectl drain, and kubectl uncordon are the standard maintenance trio. This guide walks through each command on a kubeadm lab cluster with a Deployment, DaemonSet agents, emptyDir storage, a bare Pod, and a PodDisruptionBudget.

Permanent removal—kubectl delete node, kubeadm reset, and rejoin—is covered in add, remove and rejoin cluster nodes.

Cordon vs Drain vs Uncordon

Scheduler and placement exceptions:

• Pods that tolerate node.kubernetes.io/unschedulable, including DaemonSet Pods, can still schedule on a cordoned node
• Directly setting spec.nodeName bypasses the scheduler

How each command affects running workloads:

• Cordon does not stop or move running Pods
• Drain evicts controller-managed workloads so Deployments and StatefulSets recreate replicas on other nodes
• Uncordon does not rebalance Pods that already moved; it only reopens scheduling

Prepare the Drain Lab

This lab needs at least two schedulable worker nodes:

• Evicted Pods need another worker to land on
• The cordon test needs a schedulable worker with enough capacity

Do not drain the control-plane node for this exercise. A single-machine control plane is not highly available during host maintenance.

Save the manifest below as drain-lab.yaml. It creates the drain-lab namespace, a three-replica Deployment constrained to workers, and a PodDisruptionBudget that starts with minAvailable: 3 so the PDB demonstration blocks drain until you relax it.

apiVersion: v1
kind: Namespace
metadata:
  name: drain-lab
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: drain-demo
  namespace: drain-lab
spec:
  replicas: 3
  selector:
    matchLabels:
      app: drain-demo
  template:
    metadata:
      labels:
        app: drain-demo
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: node-role.kubernetes.io/control-plane
                operator: DoesNotExist
      containers:
      - name: app
        image: registry.k8s.io/pause:3.10
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: drain-demo-pdb
  namespace: drain-lab
spec:
  minAvailable: 3
  selector:
    matchLabels:
      app: drain-demo

Apply the manifest and wait for the Deployment rollout:

kubectl apply -f drain-lab.yaml

kubectl rollout status deployment/drain-demo \
  -n drain-lab --timeout=120s

Derive the drain target from a running Deployment Pod instead of hard-coding a node name. The worker-only nodeAffinity rule guarantees every drain-demo Pod runs on a worker, so $TARGET_NODE is always a worker name:

TARGET_NODE=$(kubectl get pod -n drain-lab \
  -l app=drain-demo \
  -o jsonpath='{.items[0].spec.nodeName}')

echo "$TARGET_NODE"

Sample output:

worker01

Pin the emptyDir and unmanaged Pods on that node with spec.nodeName:

kubectl apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: emptydir-pod
  namespace: drain-lab
spec:
  nodeName: $TARGET_NODE
  containers:
  - name: app
    image: registry.k8s.io/pause:3.10
    volumeMounts:
    - name: scratch
      mountPath: /scratch
  volumes:
  - name: scratch
    emptyDir: {}
---
apiVersion: v1
kind: Pod
metadata:
  name: unmanaged-pod
  namespace: drain-lab
spec:
  nodeName: $TARGET_NODE
  containers:
  - name: app
    image: registry.k8s.io/pause:3.10
  restartPolicy: Never
EOF

Wait until both directly assigned Pods report Ready:

kubectl wait -n drain-lab \
  --for=condition=Ready \
  pod/emptydir-pod pod/unmanaged-pod \
  --timeout=60s

Sample output:

pod/emptydir-pod condition met
pod/unmanaged-pod condition met

kubectl wait supports multiple resources and waits until the condition is satisfied for each one.

Confirm the PDB reports zero allowed disruptions while minAvailable: 3 matches the replica count:

kubectl wait -n drain-lab \
  --for=jsonpath='{.status.disruptionsAllowed}'=0 \
  pdb/drain-demo-pdb \
  --timeout=60s

Sample output:

poddisruptionbudget.policy/drain-demo-pdb condition met

kubectl wait supports waiting for an exact JSONPath value.

kubectl get pdb -n drain-lab

Sample output:

NAME             MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS   AGE
drain-demo-pdb   3               N/A               0                     <age>

Inspect and Cordon the Node

List what is running on the node you plan to drain:

kubectl get pods -A \
  --field-selector "spec.nodeName=$TARGET_NODE" \
  -o wide

Sample output (trimmed to lab and node-agent Pods):

NAMESPACE       NAME                              READY   STATUS    RESTARTS   AGE    IP            NODE            NOMINATED NODE   READINESS GATES
calico-system   calico-node-<pod-suffix>          1/1     Running   0          <age>  <node-ip>     worker01        <none>           <none>
calico-system   csi-node-driver-<pod-suffix>      2/2     Running   0          <age>  <pod-ip>      worker01        <none>           <none>
drain-lab       drain-demo-<replicaset-hash>-<pod-suffix>   1/1     Running   0          <age>  <pod-ip>      worker01        <none>           <none>
drain-lab       emptydir-pod                      1/1     Running   0          <age>  <pod-ip>      worker01        <none>           <none>
drain-lab       unmanaged-pod                     1/1     Running   0          <age>  <pod-ip>      worker01        <none>           <none>
kube-system     kube-proxy-<pod-suffix>           1/1     Running   0          <age>  <node-ip>     worker01        <none>           <none>

The scheduler may place other drain-demo replicas on different nodes. At least one appears here because $TARGET_NODE was derived from a Deployment Pod.

Before you cordon or drain, note:

• Whether the Node is Ready and whether spec.unschedulable is already true
• Pod owners — Deployment, DaemonSet, bare Pod, or static/mirror Pod
• PodDisruptionBudgets that select those Pods (kubectl get pdb -A)
• Pods using emptyDir volumes
• DaemonSet Pods that drain will refuse to delete without --ignore-daemonsets

Mark the node unschedulable when you want to stop new scheduling but keep current Pods running:

kubectl cordon "$TARGET_NODE"

Sample output:

node/worker01 cordoned

Verify the status column shows SchedulingDisabled:

kubectl get nodes

Sample output:

NAME       STATUS                     ROLES           AGE    VERSION
k8s-cp     Ready                      control-plane   <age>  v1.36.3
worker01   Ready,SchedulingDisabled   <none>          <age>  v1.36.3

Existing Pods on the cordoned node keep running. Create a new Pod without a node selector to confirm scheduling skips it:

kubectl run cordon-test -n drain-lab \
  --image=registry.k8s.io/pause:3.10 \
  --restart=Never

kubectl wait -n drain-lab \
  --for=condition=Ready pod/cordon-test \
  --timeout=60s

kubectl get pod cordon-test -n drain-lab -o wide

Sample output:

NAME          READY   STATUS    RESTARTS   AGE   IP         NODE     NOMINATED NODE   READINESS GATES
cordon-test   1/1     Running   0          <age> <pod-ip>   <other-schedulable-node>   <none>           <none>

This test requires another schedulable node with enough capacity. The Pod should not use $TARGET_NODE, but its exact destination depends on the scheduler and the other eligible nodes.

Cordon marks only the selected node unschedulable; it does not restrict scheduling to worker-role nodes.

Drain the Node

Drain cordons the node (if it is not already cordoned) and evicts Pods the API can move. Modern clusters use the eviction API, which respects PodDisruptionBudgets when possible.

Uncordon first if you cordoned the node only for the scheduling test above:

kubectl uncordon "$TARGET_NODE"

Understand the Initial Blockers

Start without flags to see every blocker drain reports:

kubectl drain "$TARGET_NODE"

Without extra flags, drain stops when it hits DaemonSets, emptyDir volumes, or bare Pods. A bare kubectl drain on a typical worker reports all blockers at once:

node/worker01 cordoned
error: unable to drain node "worker01" due to error: [cannot delete DaemonSet-managed Pods (use --ignore-daemonsets to ignore): calico-system/calico-node-<pod-suffix>, calico-system/csi-node-driver-<pod-suffix>, kube-system/kube-proxy-<pod-suffix>, cannot delete Pods with local storage (use --delete-emptydir-data to override): drain-lab/emptydir-pod, kube-system/metrics-server-<replicaset-hash>-<pod-suffix>, cannot delete Pods that declare no controller (use --force to override): drain-lab/unmanaged-pod], continuing command...
There are pending nodes to be drained:
 worker01
cannot delete DaemonSet-managed Pods (use --ignore-daemonsets to ignore): calico-system/calico-node-<pod-suffix>, calico-system/csi-node-driver-<pod-suffix>, kube-system/kube-proxy-<pod-suffix>
cannot delete Pods with local storage (use --delete-emptydir-data to override): drain-lab/emptydir-pod, kube-system/metrics-server-<replicaset-hash>-<pod-suffix>
cannot delete Pods that declare no controller (use --force to override): drain-lab/unmanaged-pod

Although drain stopped before removing the workload Pods, it already cordoned the node. The node remains SchedulingDisabled; a failed drain does not automatically uncordon it. The following commands continue with the node cordoned.

Read the message as a checklist and add flags only for categories you intend to override:

• --ignore-daemonsets — first flag added in the later commands
• --delete-emptydir-data — covered in the emptyDir section
• --force — covered in the unmanaged Pod section

Eviction and warning order can vary because drain processes multiple Pods concurrently.

Handle DaemonSet Pods

Every node in this lab runs CNI and kube-proxy DaemonSets. Drain refuses to delete those Pods because the DaemonSet controller would immediately recreate them on the same node.

Supply --ignore-daemonsets so drain proceeds:

Warning: ignoring DaemonSet-managed Pods: calico-system/calico-node-<pod-suffix>, calico-system/csi-node-driver-<pod-suffix>, kube-system/kube-proxy-<pod-suffix>

Drain leaves DaemonSet Pods running on the node while evicting eligible workload Pods:

• They stop if the node is rebooted or kubelet/container networking is stopped
• See Kubernetes DaemonSets for how those controllers work

Handle emptyDir Data

Pods with emptyDir volumes store data on the node filesystem. Drain blocks eviction unless you confirm the data can be discarded.

Without --delete-emptydir-data, drain reports:

cannot delete Pods with local storage (use --delete-emptydir-data to override): drain-lab/emptydir-pod

When ephemeral scratch data is safe to lose:

• Add --delete-emptydir-data to the complete drain command when you accept losing data stored in emptyDir
• The unmanaged Pod still requires --force, which is added in the next section
• Data in emptyDir is removed when the Pod is deleted—it does not travel with the Pod to another node

Treat --delete-emptydir-data as acceptance of data loss for those volumes.

Handle Unmanaged Pods

Bare Pods created directly (no Deployment, ReplicaSet, or Job owner) block drain:

cannot delete Pods that declare no controller (use --force to override): drain-lab/unmanaged-pod

Use --force only when you understand the Pod has no controller to recreate it:

• --force is not a grace-period override for every Pod
• --force does not disable PodDisruptionBudget checks
• In this lab, --force is the final override required for the two unmanaged lab Pods

The next section runs the complete drain command with a finite timeout so you can observe the intentionally blocking PDB.

Observe PodDisruptionBudget Behaviour

A PDB limits how many Pods from a selector may be unavailable during voluntary disruption. Drain uses the eviction API, so PDB rules apply.

With minAvailable: 3 and three replicas, the budget allows no disruptions:

kubectl get pdb -n drain-lab

Sample output:

NAME             MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS   AGE
drain-demo-pdb   3               N/A               0                     <age>

Run drain with the blocker flags and a finite timeout. --timeout=0s, the default, means no timeout:

kubectl drain "$TARGET_NODE" \
  --ignore-daemonsets \
  --delete-emptydir-data \
  --force \
  --timeout=60s

Drain retries and eventually exits when the PDB blocks every eviction attempt:

evicting pod drain-lab/emptydir-pod
evicting pod drain-lab/unmanaged-pod
pod/emptydir-pod evicted
pod/unmanaged-pod evicted
evicting pod drain-lab/drain-demo-<replicaset-hash>-<pod-suffix>
error when evicting pods/"drain-demo-<replicaset-hash>-<pod-suffix>" -n "drain-lab" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
...
error: unable to drain node "worker01" due to error: [error when evicting pods/"drain-demo-<replicaset-hash>-<pod-suffix>" -n "drain-lab": global timeout reached: 1m0s, ...]

The number of drain-demo Pods shown depends on how the scheduler distributed the replicas. At least one is present because $TARGET_NODE was derived from a Deployment Pod.

Although the command ended with a timeout, it made partial progress:

• emptydir-pod and unmanaged-pod were removed because no PDB selected them
• Only the Deployment Pods remained blocked
• Drain does not roll back Pods that were already evicted

That is normal PDB protection—not a broken cluster.

Relax the budget so one Pod may leave while two remain available:

kubectl patch pdb drain-demo-pdb -n drain-lab \
  -p '{"spec":{"minAvailable":2}}'

Wait until the disruption controller reports one allowed disruption:

kubectl wait -n drain-lab \
  --for=jsonpath='{.status.disruptionsAllowed}'=1 \
  pdb/drain-demo-pdb \
  --timeout=60s

Sample output:

poddisruptionbudget.policy/drain-demo-pdb condition met

kubectl get pdb -n drain-lab

Sample output:

NAME             MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS   AGE
drain-demo-pdb   2               N/A               1                     <age>

ALLOWED DISRUPTIONS: 1 means the eviction API currently permits one additional disruption:

• kubectl drain may submit eviction requests for multiple Pods concurrently
• After one request succeeds, other requests covered by the same PDB can be rejected and retried
• Another eviction is commonly permitted after a replacement Pod becomes Ready

Single-replica workloads need extra planning:

• Draining a node that hosts the only replica can cause an availability gap while its replacement starts
• A PDB with minAvailable: 1 blocks that voluntary eviction—scale the workload or plan downtime first
• A configuration that permits the only replica to be disrupted can produce 100% temporary unavailability

Rerun drain with a longer timeout so replacements can become Ready between evictions:

kubectl drain "$TARGET_NODE" \
  --ignore-daemonsets \
  --delete-emptydir-data \
  --force \
  --timeout=300s

Sample output:

Warning: ignoring DaemonSet-managed Pods: calico-system/calico-node-<pod-suffix>, calico-system/csi-node-driver-<pod-suffix>, kube-system/kube-proxy-<pod-suffix>
evicting pod drain-lab/drain-demo-<replicaset-hash>-<pod-suffix>
pod/drain-demo-<replicaset-hash>-<pod-suffix> evicted
node/worker01 drained

A successful drain confirms that the selected Pods were evicted and removed; it does not guarantee that the Deployment has already restored all three Ready replicas. With minAvailable: 2, drain can finish after evicting the only Deployment Pod on the target while its replacement is still starting elsewhere.

Wait for the Deployment rollout to complete:

kubectl rollout status deployment/drain-demo \
  -n drain-lab --timeout=120s

Sample output:

deployment "drain-demo" successfully rolled out

kubectl rollout status waits until the Deployment rollout is complete. Then inspect the drained node and begin maintenance.

After drain succeeds, only DaemonSet (and static mirror) Pods should remain on the node:

kubectl get pods -A \
  --field-selector "spec.nodeName=$TARGET_NODE" \
  -o wide

Sample output:

NAMESPACE       NAME                    READY   STATUS    RESTARTS   AGE    IP            NODE       NOMINATED NODE   READINESS GATES
calico-system   calico-node-<pod-suffix>       1/1     Running   0          <age>  <node-ip>     worker01   <none>           <none>
calico-system   csi-node-driver-<pod-suffix>   2/2     Running   0          <age>  <pod-ip>      worker01   <none>           <none>
kube-system     kube-proxy-<pod-suffix>        1/1     Running   0          <age>  <node-ip>     worker01   <none>           <none>

A node is ready for maintenance only after kubectl drain returns successfully.

To unblock drain when PDB retries persist:

• Ensure another worker can accept evicted Pods
• Fix unhealthy replicas so the budget counts Ready Pods correctly
• Adjust the maintenance plan—scale up, relax the PDB temporarily, or drain during a window when lower availability is acceptable

--disable-eviction forces direct Pod deletion and bypasses PDB handling. Reserve it for disaster recovery, not routine maintenance. PDB design and minAvailable math live in PodDisruptionBudget.

Perform Maintenance and Uncordon

After node/<name> drained appears:

• Confirm application Pods no longer run on the node (kubectl get pods -A --field-selector "spec.nodeName=$TARGET_NODE" -o wide)
• Perform OS patches, hardware work, or kubelet troubleshooting on the host
• Verify systemctl status kubelet and the container runtime on the node
• Confirm the Node object still reports Ready when the kubelet is healthy

DaemonSet Pods remain on the node while it is cordoned—plan maintenance knowing they stop if you reboot the host or stop kubelet.

When the node is healthy again, allow scheduling:

kubectl uncordon "$TARGET_NODE"

Sample output:

node/worker01 uncordoned

kubectl get nodes

The SchedulingDisabled suffix disappears while Ready stays true.

Uncordon does not move Pods back from other nodes. New Pods—created after uncordon or scaled up by controllers—can schedule on the node again. To verify scheduling, create a test Pod with a node selector or watch the next Deployment scale-out land on the restored node.

Troubleshooting

What's Next

• Configure a Highly Available Kubernetes Control Plane with kubeadm
• Back Up and Restore Kubernetes etcd
• Kubernetes PKI, kubeconfig Users, CSR and Certificate Renewal

References

• Safely drain a node — upstream drain task
• kubectl drain reference — flags and behavior
• PodDisruptionBudget — voluntary disruption and budgets
• DaemonSet — Pods that tolerate node.kubernetes.io/unschedulable
• Volumes — emptyDir lifecycle and permanent data deletion when its Pod is removed

Summary

Node maintenance in Kubernetes is a sequence, not a single command. kubectl cordon prevents ordinary scheduler placements while leaving running workloads untouched—useful for short holds or before you are ready to evict. kubectl drain cordons and evicts API-managed Pods through the eviction API, which respects PodDisruptionBudgets and refuses to delete DaemonSet agents unless you pass --ignore-daemonsets.

The flags matter because drain surfaces real constraints:

• DaemonSets stay by design
• emptyDir data disappears with the Pod
• Bare Pods need --force
• PDBs limit how many selected Pods may be unavailable simultaneously during voluntary disruptions

When drain retries on budget violations, that is the protection working—fix capacity or replica health rather than bypassing eviction without cause.

After maintenance, kubectl uncordon reopens the node to scheduling. Workloads do not automatically flow back; only new placements consider the node. For removing a host from the cluster entirely, continue with add, remove and rejoin cluster nodes after drain completes.