Kubernetes Static Pods and Mirror Pods

When you run a static Pod, the kubelet reads a manifest from disk. You do not create that workload with kubectl apply. In this lab I put nginx on one worker, watched the mirror Pod show up in the API, and then tied that same pattern to the kubeadm control-plane manifests your cluster likely already has.

What makes a Pod static?

If you have only worked with kubectl apply before, static Pods can feel backwards at first. The kubelet owns the workload directly. Here is how that differs from a normal API Pod:

• The kubelet reads the Pod manifest from disk (or a configured URL).
• The Kubernetes scheduler does not choose a node for the Pod. The kubelet reads the local manifest and runs the Pod directly on its own node.
• The Pod is permanently bound to that kubelet and that node.
• The kubelet restarts the containers when they fail.
• Deployments, ReplicaSets, and DaemonSets do not manage it.

When the kubelet is connected to a cluster, you still see a Pod object in the API. That object is a mirror:

• kubectl can show status, events, and logs
• The API object is not where you create, update, or delete the workload
• The manifest file on the node is the source of truth

Find the static Pod manifest path

On the node where you will create the static Pod, open a root shell first. Writes under /etc/kubernetes/manifests need root privileges, and shell redirection runs before sudo could elevate cat:

sudo -i

Find where kubelet expects manifest files. On my kubeadm lab the kubelet config lives at /var/lib/kubelet/config.yaml, and the systemd unit loads it with --config:

grep staticPodPath /var/lib/kubelet/config.yaml

staticPodPath: /etc/kubernetes/manifests

On my cluster that value points at /etc/kubernetes/manifests. Your install may use a different path, so always read staticPodPath from the config file rather than assuming the kubeadm default.

Confirm the directory exists on the worker:

ls -la /etc/kubernetes/manifests/

total 8
drwxr-xr-x. 2 root root 4096 Jul 23 00:02 .
drwxr-xr-x. 5 root root 4096 Jul 23 00:02 ..

An empty directory on a worker is normal. On a control-plane node you usually see etcd and API server manifests already in place.

One file-handling rule trips people up often:

• Kubelet processes every non-hidden file under staticPodPath, regardless of extension
• A name like static-web.yaml.backup is still read as a Pod manifest
• Two files with the same metadata.name on one node can produce unpredictable behavior
• Kubernetes only ignores files whose names start with a dot (.)

Keep backups outside this directory. Do not leave .bak, .old, or copied manifests here.

Create a static Pod manifest

Write a valid Pod manifest into the configured directory on one worker. Do not run kubectl apply for this object.

Stage the file safely before kubelet picks it up:

• Write to a hidden temporary name in the same directory (for example .static-web.yaml.tmp)
• Kubelet ignores dot-prefixed files, so it will not read a half-written manifest
• mv the temp file into its final name on the same filesystem so the rename is atomic

cat >/etc/kubernetes/manifests/.static-web.yaml.tmp <<'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: static-web
  labels:
    app: static-web
spec:
  containers:
  - name: web
    image: nginx:1.27-alpine
EOF

mv /etc/kubernetes/manifests/.static-web.yaml.tmp /etc/kubernetes/manifests/static-web.yaml

Both commands exit silently on success. List the manifests directory to confirm the file is there:

ls -la /etc/kubernetes/manifests/

total 12
drwxr-xr-x. 2 root root 4096 Jul 27 19:23 .
drwxr-xr-x. 5 root root 4096 Jul 23 00:02 ..
-rw-r--r--. 1 root root  203 Jul 27 19:23 static-web.yaml

Kubelet does not react instantly. It loads the file during its next manifest check, and the default fileCheckFrequency is 20 seconds. Poll with crictl instead of assuming the Pod exists right away:

STATIC_POD_ID=""
for attempt in {1..120}; do
  STATIC_POD_ID=$(crictl pods -q --name '^static-web-')
  [[ -n "$STATIC_POD_ID" ]] && break
  sleep 1
done
[[ -n "$STATIC_POD_ID" ]] || {
  echo "Timed out waiting for static-web"
  exit 1
}

Observe the static Pod locally

Stay on the same worker and inspect the Pod through the container runtime. Nothing was submitted to the API, yet kubelet has already created the sandbox locally:

crictl pods

POD ID              CREATED             STATE               NAME                    NAMESPACE           ATTEMPT             RUNTIME
ae120fce0c2f3       5 seconds ago       Ready               static-web-worker01     default             0                   (default)
ba7f2226b40f9       About an hour ago   Ready               kube-proxy-g4df2        kube-system         0                   (default)
b18c0f71d8e36       About an hour ago   Ready               csi-node-driver-hczmm   calico-system       0                   (default)
03409f0b4ebec       About an hour ago   Ready               calico-node-j9m4s       calico-system       0                   (default)

Look at the NAME column. Kubelet already renamed the Pod to static-web-worker01 by appending the node hostname (<manifest-name>-<hostname>). List containers next:

crictl ps

CONTAINER           IMAGE               CREATED             STATE               NAME                        ATTEMPT             POD ID              POD                     NAMESPACE
4ccf8be9d3205       6769dc3a703c7       5 seconds ago       Running             web                         0                   ae120fce0c2f3       static-web-worker01     default
afa4a5a621ece       be76bac219614       About an hour ago   Running             kube-proxy                  0                   ba7f2226b40f9       kube-proxy-g4df2        kube-system
00834d9474054       eddcd7045ad69       About an hour ago   Running             csi-node-driver-registrar   0                   b18c0f71d8e36       csi-node-driver-hczmm   calico-system
940651a4874ab       4fe25fe757ce9       About an hour ago   Running             calico-csi                  0                   b18c0f71d8e36       csi-node-driver-hczmm   calico-system
052be76bbf405       c281bc67214e2       About an hour ago   Running             calico-node                 0                   03409f0b4ebec       calico-node-j9m4s       calico-system

The web container should be Running. Resolve the container ID through the static Pod sandbox so you do not pick up an unrelated web container elsewhere on the node:

STATIC_POD_ID=""
WEB_ID=""
for attempt in {1..120}; do
  STATIC_POD_ID=$(crictl pods -q --state ready --name '^static-web-' --latest)
  if [[ -n "$STATIC_POD_ID" ]]; then
    WEB_ID=$(crictl ps -q --pod "$STATIC_POD_ID" --name '^web$' --latest)
  fi
  [[ -n "$WEB_ID" ]] && break
  sleep 1
done
[[ -n "$WEB_ID" ]] || {
  echo "Timed out waiting for the static-web container"
  exit 1
}
crictl logs --tail=15 "$WEB_ID"

/docker-entrypoint.sh: Configuration complete; ready for start up
2026/07/27 13:53:28 [notice] 1#1: using the "epoll" event method
2026/07/27 13:53:28 [notice] 1#1: nginx/1.27.5
2026/07/27 13:53:28 [notice] 1#1: built by gcc 14.2.0 (Alpine 14.2.0) 
2026/07/27 13:53:28 [notice] 1#1: OS: Linux 6.12.0-211.34.1.el10_2.x86_64
2026/07/27 13:53:28 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 1024:524288
2026/07/27 13:53:28 [notice] 1#1: start worker processes
2026/07/27 13:53:28 [notice] 1#1: start worker process 30

Nginx is up on the worker. No kubectl create or kubectl apply was involved.

Inspect the mirror Pod

From a workstation with kubeconfig, list Pods across namespaces. The mirror name ends with the node hostname:

kubectl get pods -A -o wide | grep -E 'NAME|static-web'

NAMESPACE            NAME                                       READY   STATUS    RESTARTS      AGE   IP               NODE       NOMINATED NODE   READINESS GATES
default              static-web-worker01                        1/1     Running   0             22s   192.168.5.57     worker01   <none>           <none>

Kubelet created that API object so you can inspect the static Pod with kubectl. The manifest omits metadata.namespace, so the mirror lands in default. Scope mirror lookups to that namespace even when your kubeconfig context points elsewhere. Resolve the name once through the label you set in the manifest:

MIRROR_POD=""
for attempt in {1..30}; do
  MIRROR_POD=$(kubectl get pods -n default -l app=static-web -o jsonpath='{.items[0].metadata.name}' 2>/dev/null)
  [[ -n "$MIRROR_POD" ]] && break
  sleep 1
done
[[ -n "$MIRROR_POD" ]] || {
  echo "Timed out waiting for the mirror Pod"
  exit 1
}
printf '%s\n' "$MIRROR_POD"

static-web-worker01

Read the annotations that prove this Pod is a mirror of a file on disk:

kubectl get pod -n default "$MIRROR_POD" -o jsonpath='{.metadata.annotations.kubernetes\.io/config\.mirror}{"\n"}{.metadata.annotations.kubernetes\.io/config\.source}{"\n"}'

88553525878eacee78bbfae9197d46a3
file

What those values mean:

• kubernetes.io/config.mirror — hash tying the API object to the local manifest
• kubernetes.io/config.source: file — kubelet created this Pod from a file, not from the scheduler

Labels from your manifest are copied through, so app=static-web still works in selectors. Editing the mirror with kubectl edit does not change the file on the node.

Confirm the mirror is on the worker where you placed the manifest:

kubectl get pod -n default "$MIRROR_POD" -o wide

NAME                  READY   STATUS    RESTARTS   AGE   IP             NODE       NOMINATED NODE   READINESS GATES
static-web-worker01   1/1     Running   0          22s   192.168.5.57   worker01   <none>           <none>

Delete the mirror Pod

Try deleting the mirror the way you would any other Pod. Capture the original UID first so you can prove kubelet recreated the API object while the local static Pod kept running:

OLD_UID=$(kubectl get pod -n default "$MIRROR_POD" -o jsonpath='{.metadata.uid}')
kubectl delete pod -n default "$MIRROR_POD"

pod "static-web-worker01" deleted

Wait for a replacement mirror with a new UID:

NEW_UID=""
for attempt in {1..30}; do
  NEW_UID=$(kubectl get pod -n default "$MIRROR_POD" -o jsonpath='{.metadata.uid}' 2>/dev/null)
  [[ -n "$NEW_UID" && "$NEW_UID" != "$OLD_UID" ]] && break
  sleep 1
done
[[ -n "$NEW_UID" && "$NEW_UID" != "$OLD_UID" ]] || {
  echo "Timed out waiting for mirror Pod recreation"
  exit 1
}

Confirm the mirror is back:

kubectl get pods -n default -o wide | grep static-web

static-web-worker01   1/1     Running   0          6s    192.168.5.57   worker01   <none>           <none>

What you should notice:

• The mirror comes back with a fresh AGE and a new UID
• The Pod IP stays the same because the local containers never stopped
• Kubelet only recreated the API view from the manifest still on disk

kubectl delete is not how you remove a static Pod. The file under staticPodPath is what keeps the workload alive.

Modify and remove the static Pod

To change the workload, edit the manifest on the node. I added an environment variable in the example below and staged it the same hidden-temp way as before:

cat >/etc/kubernetes/manifests/.static-web.yaml.tmp <<'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: static-web
  labels:
    app: static-web
spec:
  containers:
  - name: web
    image: nginx:1.27-alpine
    env:
    - name: STATIC_DEMO
      value: "v2"
EOF

mv /etc/kubernetes/manifests/.static-web.yaml.tmp /etc/kubernetes/manifests/static-web.yaml

Wait for kubelet to reconcile the change. Poll the sandbox-scoped web container until STATIC_DEMO reads v2:

STATIC_DEMO=""
for attempt in {1..120}; do
  STATIC_POD_ID=$(crictl pods -q --state ready --name '^static-web-' --latest)
  WEB_ID=""
  if [[ -n "$STATIC_POD_ID" ]]; then
    WEB_ID=$(crictl ps -q --pod "$STATIC_POD_ID" --name '^web$' --latest)
  fi
  if [[ -n "$WEB_ID" ]]; then
    STATIC_DEMO=$(crictl exec "$WEB_ID" printenv STATIC_DEMO 2>/dev/null)
  fi
  [[ "$STATIC_DEMO" == "v2" ]] && break
  sleep 1
done
[[ "$STATIC_DEMO" == "v2" ]] || {
  echo "Timed out waiting for the updated static Pod"
  exit 1
}
printf '%s\n' "$STATIC_DEMO"

v2

Local reconciliation and mirror refresh are separate steps. The container can already report v2 while the API object still shows the old spec briefly. Wait for the mirror Pod to catch up before you read its hash:

MIRROR_VALUE=""
for attempt in {1..120}; do
  MIRROR_VALUE=$(kubectl get pod -n default "$MIRROR_POD" -o jsonpath='{.spec.containers[0].env[?(@.name=="STATIC_DEMO")].value}' 2>/dev/null)
  [[ "$MIRROR_VALUE" == "v2" ]] && break
  sleep 1
done
[[ "$MIRROR_VALUE" == "v2" ]] || {
  echo "Timed out waiting for the mirror Pod update"
  exit 1
}

The mirror Pod picks up manifest changes too. Its config hash should change, and the env value should match what you wrote on disk:

kubectl get pod -n default "$MIRROR_POD" -o jsonpath='{.metadata.annotations.kubernetes\.io/config\.hash}{"\n"}{.spec.containers[0].env[?(@.name=="STATIC_DEMO")].value}{"\n"}'

2b487a3ce9afe2244d3b35b3dd3803df
v2

To remove the static Pod entirely, delete the manifest file on the node:

rm -f /etc/kubernetes/manifests/static-web.yaml

After kubelet tears down the local Pod:

• crictl pods no longer lists static-web-* on that node
• kubectl get pods -n default no longer shows a mirror

Removing the file is the supported delete path.

Understand static Pod limitations

Static Pods are intentionally simpler than API-managed Pods. Before you choose one for an application, keep these limits in mind:

• No ConfigMaps — put config in the manifest or on the host filesystem
• No Secrets — mount host files or bake values into the manifest instead
• No ServiceAccounts — static Pods cannot use the normal service-account token flow
• No ephemeral containers — debugging must go through crictl exec or node-level tools
• No Deployment-style rollouts — you change the local file and kubelet reconciles it
• One node per file — the same manifest name on two nodes creates two independent Pods, not one shared workload

For cluster-wide agents with normal rollouts and Pod-template features, use a DaemonSet instead.

Inspect kubeadm control-plane static Pods

If you installed Kubernetes with kubeadm, you already depend on static Pods. On the control-plane node, the same staticPodPath directory holds the core components:

/etc/kubernetes/manifests/
├── etcd.yaml
├── kube-apiserver.yaml
├── kube-controller-manager.yaml
└── kube-scheduler.yaml

List them on the control plane:

ls -la /etc/kubernetes/manifests/

total 24
drwxr-xr-x. 2 root root 4096 Jul 23 00:02 .
drwxr-xr-x. 5 root root 4096 Jul 23 00:02 ..
-rw-------. 1 root root 2604 Jul 27 18:13 etcd.yaml
-rw-------. 1 root root 3877 Jul 27 18:13 kube-apiserver.yaml
-rw-------. 1 root root 3143 Jul 27 18:13 kube-controller-manager.yaml
-rw-------. 1 root root 1929 Jul 27 18:13 kube-scheduler.yaml

Kubelet watches that directory exactly the way it watched static-web.yaml on the worker. Mirror Pods land in kube-system with the node hostname suffix:

kubectl get pods -n kube-system -o wide | grep -E 'etcd|apiserver|controller-manager|scheduler'

etcd-k8s-cp                       1/1     Running   0          64m   192.168.56.108   k8s-cp     <none>           <none>
kube-apiserver-k8s-cp             1/1     Running   0          64m   192.168.56.108   k8s-cp     <none>           <none>
kube-controller-manager-k8s-cp    1/1     Running   0          64m   192.168.56.108   k8s-cp     <none>           <none>
kube-scheduler-k8s-cp             1/1     Running   0          64m   192.168.56.108   k8s-cp     <none>           <none>

Check that etcd is also file-backed. kubeadm labels control-plane static Pods with component=etcd, and the hostname suffix varies by node:

ETCD_POD=$(kubectl get pods -n kube-system -l component=etcd -o jsonpath='{.items[0].metadata.name}')
kubectl get pod -n kube-system "$ETCD_POD" -o jsonpath='{.metadata.annotations.kubernetes\.io/config\.source}{"\n"}'

file

That is the same file source you saw on the nginx mirror. etcd is not scheduler-managed.

When a control-plane component refuses to start, check these in order:

• The matching YAML under /etc/kubernetes/manifests/ on that node
• journalctl -u kubelet on the same node
• Only after that, look for Deployment-style controllers (there are none for these components)

How the components fit together is covered in Kubernetes architecture. Upgrades that rewrite these manifest files are covered in upgrade Kubernetes cluster version.

Static Pod vs DaemonSet

Choose based on who should own the workload:

• Static Pod — bootstrap pieces and node-local processes that must run even when API scheduling is unavailable (kubeadm control plane, single-node agents tied to one kubelet)
• DaemonSet — cluster-wide node agents where you want the API to handle placement, updates, and normal Pod-template features

Troubleshooting

When something does not look right, these are the mismatches I see most often in labs:

Clean up

If the demo manifest is still on the worker, remove it:

rm -f /etc/kubernetes/manifests/static-web.yaml

Leave the control-plane manifests under /etc/kubernetes/manifests/ on the control-plane node untouched.

What's Next

• Kubernetes Volumes with Practical Examples
• Kubernetes PersistentVolume and PVC with Examples
• Kubernetes StorageClass and Dynamic Volume Provisioning

References

• Static Pods
• Create static Pods
• Debugging Kubernetes nodes with crictl — inspect Pod sandboxes, containers, logs, and processes through the CRI
• Well-Known Labels, Annotations and Taints — kubernetes.io/config.hash, config.mirror, and config.source
• Kubelet Configuration (v1beta1)

Summary

A static Pod belongs to one kubelet on one node. You place a Pod YAML under staticPodPath, verify it locally with crictl, and use the API only for visibility through a mirror Pod.

Key habits to keep:

• The manifest file on the node is the source of truth
• Mirror Pods carry kubernetes.io/config.mirror and a <name>-<hostname> naming pattern
• kubectl delete on a mirror does not stop the containers
• Changing or removing the local file is how you update or delete the workload

That same file-backed model is how kubeadm runs etcd, the API server, and the other control-plane Pods.

Use static Pods when the process must stay bound to a specific kubelet. Use a DaemonSet when you need cluster-managed placement and rollouts. From here, map these control-plane manifests to the architecture of the API server and controllers, or practice upgrades that rewrite the same files safely.