Configure a Highly Available Kubernetes Control Plane with kubeadm

A highly available Kubernetes control plane keeps the API reachable when one control-plane host fails. With kubeadm, that means a stable controlPlaneEndpoint behind a load balancer, three or more control-plane nodes in a stacked-etcd topology (or an external etcd cluster), and joins where kubeadm init --upload-certs uploads the shared control-plane trust material to the temporary kubeadm-certs Secret. Each kubeadm join --control-plane downloads that material and generates the node-specific certificates required by the joining control-plane host.

Choose an HA topology

kubeadm supports two production-style layouts. Both topologies should use an odd number of etcd members for optimal quorum. An even-member cluster can operate, but adding the fourth member does not improve failure tolerance over three members.

Stacked etcd is what most kubeadm HA labs and many on-prem clusters use. Each control-plane node runs kube-apiserver, kube-controller-manager, kube-scheduler, and a local etcd member. kubeadm wires peer URLs and certificates during kubeadm init and --control-plane joins.

External etcd suits teams that can operate a dedicated etcd cluster independently from the Kubernetes control-plane hosts. You provision and secure etcd first, then point kubeadm at those endpoints. The external topology separates Kubernetes control-plane and etcd failures; it should not suggest casually reusing an etcd cluster serving unrelated systems. kubeadm does not start local etcd static Pods in that mode.

etcd is a quorum system. With n members, the cluster needs a majority ⌊n/2⌋ + 1 to accept writes. An etcd cluster with n members can tolerate floor((n - 1) / 2) failed members while retaining a majority:

• 3 members → survives 1 failure
• 5 members → survives 2 failures
• 4 members → still survives only 1 failure, but costs another host without extra tolerance

For CKA-style labs, three stacked control-plane nodes plus a load balancer is the usual target.

Plan the lab

Sketch the network before you install packages. Every node must resolve the API DNS name to the load-balancer address, not to a single control-plane IP.

Load balancer (stable API endpoint :6443)
├── control-plane-1
├── control-plane-2
└── control-plane-3

Workers
├── worker-1
└── worker-2

Example values you can copy into a planning sheet:

Prepare every control-plane and worker host with swap off, br_netfilter, containerd, and matching Kubernetes packages before you touch the load balancer. The node-prep steps in the install guide apply unchanged.

Configure the API load balancer

The load balancer terminates TCP traffic on port 6443 and forwards it to each healthy API server on the control-plane nodes. It does not terminate TLS for you in the common kubeadm pattern—the API servers speak HTTPS directly, and the balancer passes encrypted TCP through.

Requirements:

• Front-end listens on the stable address you will put in controlPlaneEndpoint (VIP or cloud LB IP) on port 6443
• Back-end targets are each control-plane node IP on port 6443
• Health checks use HTTPS against /readyz or TCP connect to 6443
• Session affinity is not required; any healthy API server can serve a given request

A minimal HAProxy TCP frontend/back-end pair looks like this:

frontend k8s-api
    bind 192.168.56.200:6443
    mode tcp
    default_backend k8s-api-back

backend k8s-api-back
    mode tcp
    balance roundrobin
    option tcp-check
    server cp-1 192.168.56.111:6443 check
    server cp-2 192.168.56.112:6443 check
    server cp-3 192.168.56.113:6443 check

The HAProxy configuration assumes 192.168.56.200 is already assigned to the load-balancer host. HAProxy does not create or move the VIP.

Keepalived does not replace HAProxy. It moves a VIP between multiple load-balancer hosts, while HAProxy forwards the API traffic. A single HAProxy host remains a single point of failure for the otherwise redundant control plane. A cloud load balancer can provide the same stable front-end address without Keepalived on your own hosts—the important part is that k8s-api.lab.local (or your chosen name) resolves to an address that always reaches a live API server.

Validate that HAProxy is listening and that the load-balancer host can reach the first control-plane address. The API /readyz check becomes meaningful only after kubeadm init starts the first API server.

Check the HAProxy configuration syntax:

sudo haproxy -c -f /etc/haproxy/haproxy.cfg

Sample output:

Configuration file is valid

Enable HAProxy and confirm it is listening on port 6443:

sudo systemctl enable --now haproxy

sudo ss -lntp | grep ':6443'

Before kubeadm init, confirm the load-balancer host can reach the first control-plane address on port 6443. A connection refusal is expected while no API server exists; a timeout indicates that the load balancer cannot reach the backend.

curl -sk -o /dev/null -w "%{http_code}\n" \
  https://192.168.56.111:6443/readyz

After kubeadm init, test the actual shared endpoint:

curl -sk -o /dev/null -w "%{http_code}\n" \
  https://k8s-api.lab.local:6443/readyz

Sample output:

200

HTTP 200 from /readyz means the API server process is ready.

Initialize the first control plane

Create a kubeadm configuration file on cp-1. Set controlPlaneEndpoint to the load-balancer host and port, match networking to your CNI, and list additional names clients use to reach the API in certSANs.

apiVersion: kubeadm.k8s.io/v1beta4
kind: ClusterConfiguration
kubernetesVersion: v1.36.3
controlPlaneEndpoint: "k8s-api.lab.local:6443"
networking:
  podSubnet: 192.168.0.0/16
  serviceSubnet: 10.96.0.0/12
apiServer:
  certSANs:
  - 192.168.56.200
etcd:
  local:
    dataDir: /var/lib/etcd
---
apiVersion: kubeadm.k8s.io/v1beta4
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 192.168.56.111
  bindPort: 6443
nodeRegistration:
  criSocket: unix:///run/containerd/containerd.sock

controlPlaneEndpoint is the address every node uses for discovery and ongoing API access. localAPIEndpoint.advertiseAddress is this node's IP on the cluster network. The local API advertise address is included in that node's certificate, while certSANs supplies additional names. The VIP is needed here only if clients connect directly to 192.168.56.200. The shared DNS name comes from controlPlaneEndpoint, and each joining node receives a certificate for its local API endpoint.

Initialize and upload control-plane certificates to the kubeadm-certs Secret so additional control-plane nodes can copy PKI material:

sudo kubeadm init --config kubeadm-config.yaml --upload-certs

On success, kubeadm prints:

• A worker join command (no --control-plane flag)
• A control-plane join command with --control-plane and --certificate-key
• The certificate key itself (save it; it expires in two hours by default)

Copy all three lines before the shell scrolls away. If you lose the certificate key, regenerate it with kubeadm init phase upload-certs --upload-certs on a healthy control-plane node while the key is still valid or after uploading a fresh one.

Configure kubectl and install CNI

Configure cluster admin access on the first control-plane node:

mkdir -p "$HOME/.kube"

cp -i /etc/kubernetes/admin.conf "$HOME/.kube/config"

chmod 600 "$HOME/.kube/config"

Install the same CNI you use in the single-control-plane lab. The Calico operator manifests and custom-resources.yaml with 192.168.0.0/16 from install Kubernetes with kubeadm apply here without changes.

Wait until the first control-plane node is Ready and core kube-system Pods are running before joining another control-plane member. A half-ready etcd ring makes the next join harder to debug.

List nodes:

kubectl get nodes

Sample output after CNI on cp-1:

NAME   STATUS   ROLES           AGE   VERSION
cp-1   Ready    control-plane   12m   v1.36.3

Join additional control-plane nodes

Prepare cp-2 and cp-3 with the same packages and sysctl settings as the first node. Join one control-plane host at a time so the first failure is easy to spot.

On a VirtualBox host with NAT and host-only interfaces, kubeadm may select the wrong interface for the API server and stacked-etcd peer URLs. Set an explicit advertise address on each join.

Join cp-2 with the control-plane join command from kubeadm init output:

sudo kubeadm join k8s-api.lab.local:6443 \
  --token <token> \
  --discovery-token-ca-cert-hash sha256:<hash> \
  --control-plane \
  --certificate-key <certificate-key> \
  --apiserver-advertise-address 192.168.56.112 \
  --cri-socket unix:///run/containerd/containerd.sock

Wait until the new node reports Ready:

kubectl wait --for=condition=Ready \
  node/cp-2 --timeout=180s

Sample output:

node/cp-2 condition met

Inspect the control-plane static Pods on the new host:

kubectl get pods -n kube-system \
  -l tier=control-plane \
  --field-selector spec.nodeName=cp-2 \
  -o wide

Sample output:

NAME                                READY   STATUS    RESTARTS   AGE   IP               NODE
etcd-cp-2                           1/1     Running   0          3m    192.168.56.112   cp-2
kube-apiserver-cp-2                 1/1     Running   0          3m    192.168.56.112   cp-2
kube-controller-manager-cp-2        1/1     Running   0          3m    192.168.56.112   cp-2
kube-scheduler-cp-2                 1/1     Running   0          3m    192.168.56.112   cp-2

Repeat for cp-3 with --apiserver-advertise-address 192.168.56.113 only after cp-2 is healthy.

After the third control-plane join, list etcd members from a control-plane node. This is the central proof that each control-plane join added one stacked-etcd member:

ETCD_POD=$(kubectl get pod -n kube-system \
  -l component=etcd \
  --field-selector spec.nodeName=cp-1 \
  -o jsonpath='{.items[0].metadata.name}')

kubectl exec -n kube-system "$ETCD_POD" -- \
  etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/healthcheck-client.crt \
  --key=/etc/kubernetes/pki/etcd/healthcheck-client.key \
  member list --write-out=table

Sample output:

+------------------+---------+-------+--------------------------+--------------------------+------------+
|        ID        | STATUS  | NAME  |        PEER ADDRS        |       CLIENT ADDRS       | IS LEARNER |
+------------------+---------+-------+--------------------------+--------------------------+------------+
| 8e9e05c52164694d | started | cp-1  | https://192.168.56.111:2380 | https://192.168.56.111:2379 |      false |
| 91bc3c398fb3c15d | started | cp-2  | https://192.168.56.112:2380 | https://192.168.56.112:2379 |      false |
| fd422379fda50e48 | started | cp-3  | https://192.168.56.113:2380 | https://192.168.56.113:2379 |      false |
+------------------+---------+-------+--------------------------+--------------------------+------------+

The kubeadm PKI layout provides the etcd health-check client certificate used by etcdctl. Confirm the API answers through the load-balancer address:

curl -sk -o /dev/null -w "%{http_code}\n" https://k8s-api.lab.local:6443/readyz

Join worker nodes

Workers use the worker join command from kubeadm init—the same API endpoint, token, and discovery hash, but without --control-plane or --certificate-key:

sudo kubeadm join k8s-api.lab.local:6443 --token <token> --discovery-token-ca-cert-hash sha256:<hash>

If the bootstrap token expired, create a new worker join line on any control-plane node:

sudo kubeadm token create --print-join-command

A correctly initialized HA cluster stores k8s-api.lab.local:6443 as controlPlaneEndpoint, so the printed command should already use that endpoint:

Sample output:

kubeadm join k8s-api.lab.local:6443 --token <token> \
  --discovery-token-ca-cert-hash sha256:<hash>

If it prints a single node IP, the output came from a different cluster or the cluster configuration is wrong. The generated join command uses the cluster's configured stable endpoint.

Confirm that basic workload scheduling and CNI initialization work:

kubectl delete pod ha-test --ignore-not-found

kubectl run ha-test \
  --image=busybox:1.36.1 \
  --restart=Never \
  --command -- sleep 3600

kubectl wait --for=condition=Ready \
  pod/ha-test --timeout=120s

Sample output:

pod/ha-test condition met

kubectl get pod ha-test -o wide

Sample output:

NAME      READY   STATUS    RESTARTS   AGE   IP           NODE
ha-test   1/1     Running   0          45s   192.168.1.5  worker-1

Confirm that the test Pod reaches Ready, receives a Pod IP, and schedules onto a worker. This verifies basic workload scheduling and CNI initialization; it does not by itself prove cross-node application connectivity.

Delete the test Pod when you are done:

kubectl delete pod ha-test --wait=false

Verify HA behavior

Three control-plane rows in kubectl get nodes are necessary but not sufficient. You also need a working load balancer, matching certificates, leader election for controllers, and a successful failover test.

Inspect node roles:

kubectl get nodes

Sample output:

NAME       STATUS   ROLES           AGE   VERSION
cp-1       Ready    control-plane   45m   v1.36.3
cp-2       Ready    control-plane   38m   v1.36.3
cp-3       Ready    control-plane   31m   v1.36.3
worker-1   Ready    <none>          18m   v1.36.3
worker-2   Ready    <none>          17m   v1.36.3

List control-plane and add-on Pods:

kubectl get pods -n kube-system -o wide

On a three-node HA cluster you should see etcd and API server Pods on each control-plane host.

Identify the current controller and scheduler leaders:

kubectl get lease \
  kube-controller-manager kube-scheduler \
  -n kube-system \
  -o custom-columns='NAME:.metadata.name,HOLDER:.spec.holderIdentity,RENEWED:.spec.renewTime,TRANSITIONS:.spec.leaseTransitions'

Sample output:

NAME                      HOLDER                                        RENEWED                    TRANSITIONS
kube-controller-manager   cp-2_8f3a1c2d-4e5b-6a7c-8d9e-0f1a2b3c4d5e      2026-07-27T14:55:12Z       1
kube-scheduler            cp-1_1a2b3c4d-5e6f-7a8b-9c0d-1e2f3a4b5c6d      2026-07-27T14:55:14Z       0

Controller-manager and scheduler leader election uses Lease objects, including holder, renewal time, and transition count.

Start a watch and press Ctrl+C after the holder changes:

kubectl get lease \
  kube-controller-manager kube-scheduler \
  -n kube-system --watch

Power off a control-plane VM that currently holds one of those leases. Static control-plane manifests are reconciled by the kubelet from /etc/kubernetes/manifests, so stopping an API server container with crictl stop normally causes the kubelet to recreate it. A full VM failure gives a clearer failover window.

From the workstation, retry through the load balancer until a surviving API server responds. One request can fail while the load balancer detects the dead backend; do not expect mathematically zero interruption.

until kubectl --request-timeout=5s \
  get --raw=/readyz >/dev/null 2>&1; do
  sleep 2
done

kubectl get --raw=/readyz

Sample output:

ok

Prove that etcd still accepts writes:

kubectl create configmap ha-write-check \
  --from-literal=result=write-succeeded \
  --dry-run=client -o yaml |
kubectl apply -f -

kubectl get configmap ha-write-check \
  -o jsonpath='{.data.result}{"\n"}'

Sample output:

configmap/ha-write-check created
write-succeeded

A read-only kubectl get nodes check is insufficient. The ConfigMap operation proves that the surviving API servers can still commit a write through the remaining etcd quorum.

Power the failed node back on and confirm etcd and API server Pods return to Running.

Understand the external-etcd topology

When etcd runs outside the control-plane nodes, replace the etcd.local block in ClusterConfiguration with client endpoints and TLS files:

apiVersion: kubeadm.k8s.io/v1beta4
kind: ClusterConfiguration
kubernetesVersion: v1.36.3
controlPlaneEndpoint: "k8s-api.lab.local:6443"
etcd:
  external:
    endpoints:
    - https://etcd-1.lab.local:2379
    - https://etcd-2.lab.local:2379
    - https://etcd-3.lab.local:2379
    caFile: /etc/kubernetes/pki/etcd/ca.crt
    certFile: /etc/kubernetes/pki/apiserver-etcd-client.crt
    keyFile: /etc/kubernetes/pki/apiserver-etcd-client.key

Operational differences from stacked etcd:

• The etcd cluster must exist, be secured with TLS, and pass health checks before kubeadm init
• Client certificates referenced in caFile, certFile, and keyFile must be present on every control-plane node
• kubeadm does not create local etcd static Pods in this mode
• Backup, restore, defragmentation, and member add/remove are your etcd procedures—not kubeadm phases

Snapshot and restore workflows belong in back up and restore Kubernetes etcd. The same PKI and endpoint discipline applies whether etcd is stacked or external.

Regenerate expired join material

Bootstrap tokens expire (default 24 hours). Certificate keys from --upload-certs expire faster (default two hours).

Upload a new certificate bundle for control-plane joins:

sudo kubeadm init phase upload-certs --upload-certs

Copy the new key, then run:

sudo kubeadm token create \
  --print-join-command \
  --certificate-key <new-certificate-key>

Sample output:

kubeadm join k8s-api.lab.local:6443 --token <token> \
  --discovery-token-ca-cert-hash sha256:<hash> \
  --control-plane \
  --certificate-key <new-certificate-key>

For worker joins only, omit --certificate-key:

sudo kubeadm token create --print-join-command

Common HA setup failures

Manifest and API checks with curl are covered in the curl command guide. | Join hangs on discovery | controlPlaneEndpoint DNS does not resolve from worker | Fix DNS or /etc/hosts; ping the name from each node | | x509 certificate valid for … not <name> | Missing certSANs entry for VIP or DNS | Re-init is required if the first API cert lacks SANs; plan SANs before kubeadm init | | certificate key expired on control-plane join | --certificate-key older than two hours | Run kubeadm init phase upload-certs --upload-certs and join again with the new key | | Nodes NotReady after join | CNI not installed or Pod CIDR mismatch | Install Calico (or your CNI) with the same podSubnet as ClusterConfiguration | | etcd pod crash loop on new control-plane | Firewall blocks peer port 2380 or wrong advertise URLs | Open 2379/2380 between control-plane nodes; check etcd Pod logs with crictl logs | | upload-certs x509 error mentioning an unexpected IP | Multi-homed node; kubeadm client uses an address not in API cert | Include the correct address in certSANs, or route API traffic only through the cluster interface | | All control planes joined at once; cluster broken | First join error hidden by parallel joins | Join control-plane nodes one at a time; verify etcd and API after each |

What's Next

• Back Up and Restore Kubernetes etcd
• Kubernetes PKI, kubeconfig Users, CSR and Certificate Renewal
• Upgrade a Kubernetes Cluster with kubeadm

References

• Creating Highly Available Clusters with kubeadm — upstream stacked and external etcd procedures
• Options for Highly Available Topology — stacked versus external-etcd behavior
• kubeadm configuration (v1beta4) — controlPlaneEndpoint, certSANs, and etcd.external
• kubeadm token — generating worker and control-plane join commands
• PKI certificates and requirements — etcd health-check client certificates
• Operating etcd clusters for Kubernetes — member and endpoint health verification
• Leases — controller-manager and scheduler leader election
• etcd FAQ — Why an odd number of members?

Summary

Highly available kubeadm clusters center on one stable API address: controlPlaneEndpoint behind a TCP load balancer on port 6443, with certSANs that cover the VIP when clients connect directly to it. Stacked etcd on three control-plane nodes is the default teaching path—each join after the first uses --control-plane and a fresh --certificate-key from --upload-certs.

You installed CNI and waited for a healthy first control plane before expanding etcd, joined workers through the same endpoint without control-plane flags, and validated HA by checking Lease holders and surviving the loss of one control-plane VM with a write test through the remaining etcd quorum. External etcd swaps the local etcd block for etcd.external endpoints and shifts backup and member operations to your etcd team.

The usual production pitfalls are certificate SAN gaps, expired certificate keys, and load balancers that never forward 6443. Join control-plane nodes sequentially, keep token and certificate-key material current, and run a deliberate failover test before you call the cluster highly available.