Back Up and Restore Kubernetes etcd

Kubernetes stores cluster state served by kube-apiserver in etcd, including:

• namespaces, workloads, and RBAC objects
• CRDs and custom resources

Data served by aggregated API servers can be stored elsewhere. If you lose the etcd data directory on a kubeadm control plane without a snapshot, you lose that API state. This walkthrough takes a live snapshot with etcdctl, verifies it with etcdutl, restores into a new directory, and proves the restore by bringing back objects that no longer exist after you deleted them.

Identify the etcd topology

Before you run any client command, confirm how etcd is deployed and where the certificates and data live.

For a kubeadm stacked topology, etcd runs as a static Pod. Read the manifest rather than guessing paths:

grep -E -- \
  '--(name|data-dir|initial-advertise-peer-urls|initial-cluster)=|path: /var/lib/etcd' \
  /etc/kubernetes/manifests/etcd.yaml

On this lab the important lines are:

- --data-dir=/var/lib/etcd
    - --initial-advertise-peer-urls=https://192.168.56.108:2380
    - --initial-cluster=k8s-cp=https://192.168.56.108:2380
    - --name=k8s-cp
      path: /var/lib/etcd

Record the member name, peer URL, and initial cluster string now. You need them again during etcdutl snapshot restore:

ETCD_NAME=$(
  awk -F= '/--name=/{print $2; exit}' \
    /etc/kubernetes/manifests/etcd.yaml
)

ETCD_PEER_URL=$(
  awk -F= '/--initial-advertise-peer-urls=/{print $2; exit}' \
    /etc/kubernetes/manifests/etcd.yaml
)

ETCD_INITIAL_CLUSTER=$(
  awk -F'--initial-cluster=' '/--initial-cluster=/{print $2; exit}' \
    /etc/kubernetes/manifests/etcd.yaml | awk '{print $1}'
)

printf 'name=%s\npeer=%s\ncluster=%s\n' \
  "$ETCD_NAME" "$ETCD_PEER_URL" "$ETCD_INITIAL_CLUSTER"

Sample output:

name=k8s-cp
peer=https://192.168.56.108:2380
cluster=k8s-cp=https://192.168.56.108:2380

These membership values define the new logical cluster created from the snapshot.

Record these values before you continue:

If ClusterConfiguration uses etcd.external:

• back up that external cluster with its own endpoints and certificates
• use the same snapshot commands; the static Pod steps below apply only to stacked etcd

HA topology choices are covered in configure a highly available control plane with kubeadm.

Prepare the etcd client environment

Install etcdctl and etcdutl that match the major.minor of the etcd image (here 3.6.8). kubeadm does not install those binaries on the host by default. Download the official release tarball for your architecture, then place both tools on PATH:

ETCD_VER=v3.6.8
ARCH=amd64

curl -L \
  "https://github.com/etcd-io/etcd/releases/download/${ETCD_VER}/etcd-${ETCD_VER}-linux-${ARCH}.tar.gz" \
  -o /tmp/etcd.tar.gz

tar -xzf /tmp/etcd.tar.gz -C /tmp

install -m 0755 \
  "/tmp/etcd-${ETCD_VER}-linux-${ARCH}/etcdctl" \
  "/tmp/etcd-${ETCD_VER}-linux-${ARCH}/etcdutl" \
  /usr/local/bin/

On ARM64 systems, set ARCH=arm64 before the download.

Confirm versions:

etcdctl version

Sample output values such as timestamps, revisions, hashes, member IDs, database sizes, response times, and object ages vary between clusters.

Sample output:

etcdctl version: 3.6.8
API version: 3.6

etcdutl version

Sample output:

etcdutl version: 3.6.8
API version: 3.6

Set client variables from the manifest. Certificate choices on a kubeadm control plane:

• healthcheck-client — preferred for host-side etcdctl; built for probes and admin checks
• server pair — also works as a client cert when the CA trusts it
• apiserver-etcd-client — reserved for kube-apiserver; do not use as your shell identity

export ETCDCTL_ENDPOINTS=https://127.0.0.1:2379

export ETCDCTL_CACERT=/etc/kubernetes/pki/etcd/ca.crt

export ETCDCTL_CERT=/etc/kubernetes/pki/etcd/healthcheck-client.crt

export ETCDCTL_KEY=/etc/kubernetes/pki/etcd/healthcheck-client.key

ETCDCTL_API=3 is unnecessary on current etcd 3.6 clients—the tool already speaks the v3 API and may warn if you set that variable.

List the PKI files so you know what you are using:

ls -la /etc/kubernetes/pki/etcd/

Sample output:

total 40
drwxr-xr-x. 2 root root 4096 Jul 24 13:54 .
drwxr-xr-x. 3 root root 4096 Jul 24 13:54 ..
-rw-r--r--. 1 root root 1094 Jul 24 13:54 ca.crt
-rw-------. 1 root root 1679 Jul 24 13:54 ca.key
-rw-r--r--. 1 root root 1123 Jul 24 13:54 healthcheck-client.crt
-rw-------. 1 root root 1675 Jul 24 13:54 healthcheck-client.key
-rw-r--r--. 1 root root 1192 Jul 24 13:54 peer.crt
-rw-------. 1 root root 1675 Jul 24 13:54 peer.key
-rw-r--r--. 1 root root 1192 Jul 24 13:54 server.crt
-rw-------. 1 root root 1675 Jul 24 13:54 server.key

Check etcd health before backup

A snapshot of an unhealthy member can still be better than nothing, but you should know the health state when you take it.

etcdctl endpoint health --write-out=table

Sample output:

+------------------------+--------+-------------+-------+
|        ENDPOINT        | HEALTH |    TOOK     | ERROR |
+------------------------+--------+-------------+-------+
| https://127.0.0.1:2379 |   true | 24.935635ms |       |
+------------------------+--------+-------------+-------+

HEALTH true means the member accepted a proposal. Next, read membership and size details:

etcdctl endpoint status --write-out=table

Sample output (trimmed columns for readability):

+------------------------+------------------+---------+---------+-----------+------------+
|        ENDPOINT        |        ID        | VERSION | DB SIZE | IS LEADER | RAFT INDEX |
+------------------------+------------------+---------+---------+-----------+------------+
| https://127.0.0.1:2379 | 73f02e3a4230ca03 |   3.6.8 |   74 MB |      true |     739578 |
+------------------------+------------------+---------+---------+-----------+------------+

Record before the snapshot:

• member ID, etcd version, and approximate database size from endpoint status
• snapshot hash and revision from etcdutl snapshot status after snapshot save

After recovery, prove correctness through restored API objects rather than expecting revision equality.

Create test cluster state

Create a few API objects so you can prove the restore later. Application files on PersistentVolumes are not in etcd—this restore only brings back Kubernetes API state.

kubectl create namespace etcd-restore-lab

kubectl create configmap etcd-restore-marker -n etcd-restore-lab --from-literal=phase=before-snapshot

kubectl create deployment etcd-restore-demo \
  -n etcd-restore-lab \
  --image=registry.k8s.io/e2e-test-images/agnhost:2.45 \
  -- /agnhost pause

pause is the documented agnhost subcommand for keeping the container running:

• pass the full /agnhost pause path in the Deployment command
• a bare pause argument fails because the image entrypoint is not on PATH

Wait until the Deployment reports available:

kubectl wait --for=condition=Available deployment/etcd-restore-demo -n etcd-restore-lab --timeout=120s

Sample output:

deployment.apps/etcd-restore-demo condition met

Leave these objects in place until after snapshot save finishes.

Save an etcd snapshot

etcdctl snapshot save connects to the running etcd server and writes a point-in-time database file. Store it outside /var/lib/etcd so a later restore cannot overwrite the only copy.

mkdir -p /var/backups

etcdctl snapshot save /var/backups/etcd-snapshot-lab.db

Sample output:

{"level":"info","ts":"2026-07-27T17:45:04.868842+0530","caller":"snapshot/v3_snapshot.go:83","msg":"created temporary db file","path":"/var/backups/etcd-snapshot-lab.db.part"}
{"level":"info","ts":"2026-07-27T17:45:05.032463+0530","caller":"snapshot/v3_snapshot.go:96","msg":"fetching snapshot","endpoint":"https://127.0.0.1:2379"}
{"level":"info","ts":"2026-07-27T17:45:12.619409+0530","caller":"snapshot/v3_snapshot.go:121","msg":"saved","path":"/var/backups/etcd-snapshot-lab.db"}
Snapshot saved at /var/backups/etcd-snapshot-lab.db
Server version 3.6.0

Confirm size and permissions, then checksum the file before you copy it off the node:

ls -lh /var/backups/etcd-snapshot-lab.db

Sample output:

-rw-------. 1 root root 72M Jul 27 17:45 /var/backups/etcd-snapshot-lab.db

sha256sum /var/backups/etcd-snapshot-lab.db

Sample output:

e86ee6d4457043c3ad65160cc48c52362b118b958b7b5c225cace105dfac1d61  /var/backups/etcd-snapshot-lab.db

Expected snapshot file traits:

• mode 600 and root ownership
• copy the file to another host or object storage before you practice destructive restores

Verify snapshot status

Offline inspection uses etcdutl, not the live endpoint:

etcdutl snapshot status /var/backups/etcd-snapshot-lab.db --write-out=table

Sample output:

+----------+----------+------------+------------+---------+
|   HASH   | REVISION | TOTAL KEYS | TOTAL SIZE | VERSION |
+----------+----------+------------+------------+---------+
| 1da9612b |   574643 |        770 |      74 MB |   3.6.0 |
+----------+----------+------------+------------+---------+

Record hash, revision, key count, and size for your backup log. etcdctl snapshot status is deprecated; prefer etcdutl.

Diverge the live cluster from the snapshot

Delete the lab namespace and create a marker that exists only after the backup. A successful restore must:

• bring the etcd-restore-lab namespace back
• remove the post-snapshot-marker ConfigMap from default

kubectl delete namespace etcd-restore-lab --wait=true

kubectl create configmap post-snapshot-marker -n default --from-literal=phase=after-snapshot

Confirm the live cluster no longer has the lab namespace and does have the post-snapshot ConfigMap:

kubectl get ns etcd-restore-lab

Sample output:

Error from server (NotFound): namespaces "etcd-restore-lab" not found

kubectl get cm post-snapshot-marker -n default

Sample output:

NAME                   DATA   AGE
post-snapshot-marker   1      0s

Restore the snapshot to a new data directory

Stop writing through the API and stop the etcd static Pod before you restore. Move the manifests out of /etc/kubernetes/manifests so kubelet tears the Pods down. Keep copies of the original files.

cp -a /etc/kubernetes/manifests/etcd.yaml /root/etcd.yaml.lab

cp -a /etc/kubernetes/manifests/kube-apiserver.yaml /root/kube-apiserver.yaml.lab

mv /etc/kubernetes/manifests/kube-apiserver.yaml /tmp/kube-apiserver.yaml.off

mv /etc/kubernetes/manifests/etcd.yaml /tmp/etcd.yaml.off

Wait until the etcd and kube-apiserver containers stop. crictl pods can still list Pod sandboxes after the containers exit, so check running containers with bounded retries. A crictl failure must not be treated as an empty result—if the runtime endpoint is unreachable, exit and fix containerd before continuing.

stopped=false

for attempt in {1..30}; do
  etcd_id=$(crictl ps -q --name '^etcd$') || {
    echo "crictl could not query the container runtime"
    exit 1
  }
  apiserver_id=$(crictl ps -q --name '^kube-apiserver$') || {
    echo "crictl could not query the container runtime"
    exit 1
  }

  if [[ -z "$etcd_id" && -z "$apiserver_id" ]]; then
    echo "etcd and kube-apiserver stopped"
    stopped=true
    break
  fi

  sleep 2
done

[[ "$stopped" == true ]] || {
  echo "Timed out waiting for etcd and kube-apiserver to stop"
  exit 1
}

Sample output:

etcd and kube-apiserver stopped

Restore into a new directory. Overwriting /var/lib/etcd in place while anything still holds the DB open is an easy way to lose both the live data and the restore:

etcdutl snapshot restore /var/backups/etcd-snapshot-lab.db \
  --data-dir=/var/lib/etcd-from-backup \
  --name="${ETCD_NAME}" \
  --initial-cluster="${ETCD_INITIAL_CLUSTER}" \
  --initial-cluster-token=etcd-cluster-restore-1 \
  --initial-advertise-peer-urls="${ETCD_PEER_URL}" \
  --bump-revision=1000000000 \
  --mark-compacted

Sample output (trimmed):

2026-07-27T17:48:25+05:30	info	snapshot/v3_snapshot.go:305	restoring snapshot	{"path": "/var/backups/etcd-snapshot-lab.db", "data-dir": "/var/lib/etcd-from-backup"}
2026-07-27T17:48:26+05:30	info	membership/cluster.go:424	added member	{"cluster-id": "1d9dda91848dcee7", "added-peer-id": "a8ecca4e4d27075d", "added-peer-peer-urls": ["https://192.168.56.108:2380"]}
2026-07-27T17:48:26+05:30	info	snapshot/v3_snapshot.go:333	restored snapshot	{"path": "/var/backups/etcd-snapshot-lab.db", "data-dir": "/var/lib/etcd-from-backup"}

Restore flags that matter on Kubernetes:

• --name and peer URLs must match what the etcd static Pod will advertise
• --initial-cluster-token starts a new logical cluster from the snapshot
• --bump-revision prevents the restored Kubernetes revision from moving backward
• --mark-compacted forces watch clients to discard stale state and perform a fresh list operation

The restore finishes with a ready data directory under /var/lib/etcd-from-backup.

Point the kubeadm static Pod to restored data

On kubeadm stacked etcd:

• the container always sees data at /var/lib/etcd because that is the volume mountPath
• you change only the hostPath so the restored files bind-mount to that path
• leave --data-dir=/var/lib/etcd unchanged inside the Pod spec

Edit the backed-up manifest so the etcd-data volume points at the restore directory, then install it:

sed 's|path: /var/lib/etcd$|path: /var/lib/etcd-from-backup|' /root/etcd.yaml.lab | tee /etc/kubernetes/manifests/etcd.yaml >/dev/null

Confirm the two critical lines disagree on purpose—container path vs host path:

grep -E 'data-dir|path: /var/lib' /etc/kubernetes/manifests/etcd.yaml

Sample output:

- --data-dir=/var/lib/etcd
      path: /var/lib/etcd-from-backup

Wait for etcd to become healthy again:

etcd_ready=false

for attempt in {1..60}; do
  if etcdctl endpoint health >/dev/null 2>&1; then
    etcd_ready=true
    break
  fi

  sleep 2
done

[[ "$etcd_ready" == true ]] || {
  echo "Timed out waiting for etcd"
  exit 1
}

etcdctl endpoint health --write-out=table

Sample output:

+------------------------+--------+-------------+-------+
|        ENDPOINT        | HEALTH |    TOOK     | ERROR |
+------------------------+--------+-------------+-------+
| https://127.0.0.1:2379 |   true | 42.893976ms |       |
+------------------------+--------+-------------+-------+

Then restore the API server manifest:

cp -a /root/kube-apiserver.yaml.lab /etc/kubernetes/manifests/kube-apiserver.yaml

kubelet recreates kube-apiserver. Wait for /readyz before you trust kubectl:

api_ready=false

for attempt in {1..60}; do
  if kubectl --request-timeout=3s \
    get --raw='/readyz' >/dev/null 2>&1; then
    api_ready=true
    break
  fi

  sleep 2
done

[[ "$api_ready" == true ]] || {
  echo "Timed out waiting for kube-apiserver"
  exit 1
}

Verify the restored cluster

Check API readiness:

kubectl get --raw='/readyz?verbose' | head -n 12

Sample output:

[+]ping ok
[+]log ok
[+]etcd ok
[+]etcd-readiness ok
[+]informer-sync ok
[+]poststarthook/start-apiserver-admission-initializer ok
[+]poststarthook/generic-apiserver-start-informers ok
[+]poststarthook/priority-and-fairness-config-consumer ok
[+]poststarthook/priority-and-fairness-filter ok
[+]poststarthook/storage-object-count-tracker-hook ok
[+]poststarthook/start-apiextensions-informers ok
[+]poststarthook/start-apiextensions-controllers ok

etcd ok and etcd-readiness ok mean the API server is talking to the restored member.

Confirm nodes and the pre-snapshot objects returned:

kubectl get nodes

Sample output:

NAME       STATUS   ROLES           AGE    VERSION
k8s-cp     Ready    control-plane   3d3h   v1.36.3
worker01   Ready    <none>          158m   v1.36.3

kubectl get ns etcd-restore-lab

Sample output:

NAME               STATUS   AGE
etcd-restore-lab   Active   6m25s

kubectl get cm etcd-restore-marker -n etcd-restore-lab

Sample output:

NAME                  DATA   AGE
etcd-restore-marker   1      6m26s

Confirm the post-snapshot ConfigMap is gone:

kubectl get cm post-snapshot-marker -n default

Sample output:

Error from server (NotFound): configmaps "post-snapshot-marker" not found

Primary recovery proof:

• etcd-restore-marker and the etcd-restore-lab namespace return
• post-snapshot-marker in default is gone

A restore also brings back the Deployment's existing .status, including an earlier Available=True condition. That means:

• kubectl wait --for=condition=Available can succeed immediately from stale status
• it does not prove the controller reconciled new Pods

Force a new generation after restore:

kubectl scale deployment/etcd-restore-demo \
  -n etcd-restore-lab \
  --replicas=2

Sample output:

deployment.apps/etcd-restore-demo scaled

Then wait for the controller to process that new state:

kubectl rollout status deployment/etcd-restore-demo \
  -n etcd-restore-lab \
  --timeout=120s

Sample output:

deployment "etcd-restore-demo" successfully rolled out

Finally confirm the Deployment and its Pods:

kubectl get deployment,pods \
  -n etcd-restore-lab \
  -l app=etcd-restore-demo

Sample output:

NAME                                READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/etcd-restore-demo   2/2     2            2           8m12s

NAME                                     READY   STATUS    RESTARTS   AGE
pod/etcd-restore-demo-5d95844cd8-7k2mf   1/1     Running   0          42s
pod/etcd-restore-demo-5d95844cd8-q9xwn   1/1     Running   0          42s

That proves new API writes work and the Deployment controller resumed reconciliation.

Re-check etcd health one more time after the API is up:

etcdctl endpoint status --write-out=table

Sample output (trimmed):

+------------------------+------------------+---------+---------+-----------+
|        ENDPOINT        |        ID        | VERSION | DB SIZE | IS LEADER |
+------------------------+------------------+---------+---------+-----------+
| https://127.0.0.1:2379 | a8ecca4e4d27075d |   3.6.8 |   74 MB |      true |
+------------------------+------------------+---------+---------+-----------+

Member ID differs from the pre-restore ID because snapshot restore builds a new cluster from the snapshot.

Restore considerations for HA etcd

A multi-member etcd restore is a cluster recovery, not a rolling single-node replacement.

• Restore every member from the same snapshot into new data directories, then start them as a new cluster with consistent member names and peer URLs from your recovery plan.
• Do not restore one outdated member into a healthy active etcd cluster—that member's history will not match the quorum.
• Quorum math still applies: you need a majority of healthy members after recovery.
• Stop all API servers before you rebuild etcd members, then bring API servers back when etcd is healthy.

Keep the full HA procedure for a dedicated disaster-recovery window. The single-member lab above is the exam-style workflow and the foundation for understanding those larger steps. Topology planning for multiple control-plane nodes is in the HA kubeadm guide.

Common restore failures

What's Next

• Kubernetes PKI, kubeconfig Users, CSR and Certificate Renewal
• Upgrade a Kubernetes Cluster with kubeadm
• Kubernetes RBAC with Examples

References

• Operating etcd clusters for Kubernetes — backup, restore, and etcdctl / etcdutl roles
• etcd v3.6 disaster recovery — revision bumps, --mark-compacted, and snapshot restore
• Save an etcd database — etcdctl snapshot save with cluster certificates
• Install etcd tools — pre-built etcdctl and etcdutl binaries
• PKI certificates and requirements — etcd health-check client certificates
• Create static Pods — kubeadm control-plane manifest behavior
• Debugging Kubernetes nodes with crictl — configure the runtime endpoint and inspect running control-plane containers

Summary

You identified stacked etcd from the kubeadm static Pod manifest, recorded member name and peer URLs for restore, pointed etcdctl at https://127.0.0.1:2379 with trusted certificates, and captured a snapshot outside the live data directory. etcdutl snapshot status recorded hash, revision, and size before you changed the cluster on purpose.

The restore path that works on kubeadm is deliberate: stop the API server and etcd, run etcdutl snapshot restore into a new host directory with --bump-revision and --mark-compacted, keep --data-dir=/var/lib/etcd inside the container, and retarget only the etcd-data hostPath. Changing --data-dir alone creates an empty database and looks like a mysterious failed restore.

After the API returned, pre-snapshot objects came back and post-snapshot writes disappeared. Scaling the lab Deployment and waiting for rollout confirmed controllers resumed reconciliation. For HA, treat restore as a full-cluster recovery with consistent membership; practice the single-member flow until the hostPath lesson is muscle memory, then plan multi-member recovery separately.