Build Immutable Containers in Kubernetes

Immutable containers are replaced, not patched. This walkthrough takes nginx from a digest-pinned baseline through a deliberate readOnlyRootFilesystem failure, inventories the paths it must write, mounts explicit emptyDir volumes for those paths, and layers non-root execution with dropped capabilities on a kubeadm lab running Kubernetes 1.36.3.

Define runtime immutability

Runtime immutability in Kubernetes combines several controls. No single field delivers it alone:

• Image digest — deploy image@sha256:… so the cluster pulls exact bytes, not a movable tag
• Read-only root filesystem — readOnlyRootFilesystem: true blocks writes to the image layer
• Explicit writable mounts — route /tmp, cache, PID files, and uploads to emptyDir or PVC
• Non-root execution — runAsNonRoot with a numeric UID and GID the image supports
• Reduced capabilities — drop ALL, add back only what the process truly needs
• Disabled privilege escalation — allowPrivilegeEscalation: false on non-privileged workloads
• Replacement-based deployment — change configuration through new Pods, not kubectl exec edits inside running containers

Writable volumes are part of the design, not a bypass. The goal is that nobody can modify the shipped image layer or install packages at runtime; ephemeral state lives on mounts you declare.

Create the lab namespace

Work in an isolated namespace so failed read-only experiments do not affect production workloads:

kubectl create namespace immutable-lab --dry-run=client -o yaml | kubectl apply -f -

namespace/immutable-lab created

Pin the image digest

Tags such as :latest or :1.27 can move at the registry. Pin the manifest digest in your Deployment:

image: quay.io/nginx/nginx-unprivileged@sha256:65e3e85dbaed8ba248841d9d58a899b6197106c23cb0ff1a132b7bfe0547e4c0

This lab uses quay.io/nginx/nginx-unprivileged because Docker Hub rate limits blocked nginx pulls during testing. The same workflow applies to any registry when you record the digest after build or promotion.

Deploy a baseline Deployment without read-only mode so you can confirm the image starts cleanly:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-baseline
  namespace: immutable-lab
  labels:
    app.kubernetes.io/name: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: web-baseline
  template:
    metadata:
      labels:
        app: web-baseline
    spec:
      containers:
        - name: nginx
          image: quay.io/nginx/nginx-unprivileged@sha256:65e3e85dbaed8ba248841d9d58a899b6197106c23cb0ff1a132b7bfe0547e4c0
          ports:
            - containerPort: 8080

kubectl apply -f baseline.yaml

kubectl -n immutable-lab rollout status deployment/web-baseline --timeout=120s

deployment "web-baseline" successfully rolled out

After the Pod is running, read imageID from status. It should match the digest in your manifest:

kubectl -n immutable-lab get pod -l app=web-baseline -o jsonpath='{.items[0].status.containerStatuses[0].imageID}{"\n"}'

quay.io/nginx/nginx-unprivileged@sha256:65e3e85dbaed8ba248841d9d58a899b6197106c23cb0ff1a132b7bfe0547e4c0

The reference in spec.containers[].image and the resolved imageID should agree on the sha256 value. Record that digest in your release record alongside Cosign verification or Trivy scan results when you promote through CI.

Enable read-only root filesystem

Add readOnlyRootFilesystem: true on the container before you add writable volumes. You want to see the real startup failure:

securityContext:
  readOnlyRootFilesystem: true

Apply a second Deployment that only sets read-only mode:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-readonly
  namespace: immutable-lab
spec:
  replicas: 1
  selector:
    matchLabels:
      app: web-readonly
  template:
    metadata:
      labels:
        app: web-readonly
    spec:
      containers:
        - name: nginx
          image: quay.io/nginx/nginx-unprivileged@sha256:65e3e85dbaed8ba248841d9d58a899b6197106c23cb0ff1a132b7bfe0547e4c0
          ports:
            - containerPort: 8080
          securityContext:
            readOnlyRootFilesystem: true

kubectl apply -f readonly-fail.yaml

Wait for the kubelet to restart the container, then inspect Pod status:

kubectl -n immutable-lab get pods -l app=web-readonly

NAME                           READY   STATUS   RESTARTS   AGE
web-readonly-77dcd4d59-76sv5   0/1     Error    2          20s

Read the container log. nginx fails while creating a temporary directory under /tmp:

kubectl -n immutable-lab logs \
  -l app=web-readonly \
  --previous \
  --tail=5

/docker-entrypoint.sh: Configuration complete; ready for start up
2026/07/28 07:37:39 [emerg] 1#1: mkdir() "/tmp/proxy_temp" failed (30: Read-only file system)
nginx: [emerg] mkdir() "/tmp/proxy_temp" failed (30: Read-only file system)

The Pod enters CrashLoopBackOff because the process cannot write anywhere the image layer does not already provide. That log line is your inventory starting point.

Inventory writable paths

Before you mount volumes, list every path the application writes at startup or steady state:

Before you mount volumes, verify the effective configuration from the working baseline:

kubectl -n immutable-lab exec deploy/web-baseline -- \
  sh -c 'nginx -T 2>&1 | grep -E "^[[:space:]]*(pid|.*_temp_path)"'

For the pinned nginx-unprivileged image, the PID file and default temporary directories live under /tmp. Run the application once without read-only mode and trace strace, startup scripts, or documentation when you use a custom NGINX configuration. For this lab, the /tmp/proxy_temp error already names the first missing mount.

Add dedicated emptyDir mounts

Mount one emptyDir volume per writable path the process needs. Do not mount a single large writable root; declare each path explicitly so reviewers and scanners can see the contract:

volumeMounts:
  - name: tmp
    mountPath: /tmp
volumes:
  - name: tmp
    emptyDir: {}

For this pinned nginx-unprivileged image, /tmp is sufficient. Add cache or run mounts only when nginx -T or your custom configuration proves they are required.

emptyDir is backed by node storage (or memory when you set medium: Memory). Data disappears when the Pod is deleted. That is correct for cache, PID files, and temporary uploads you do not need to keep.

The non-root nginx image runs as UID 101. In this tested Pod, UID 101 can write to the mounted emptyDir. Verify ownership and mode for your runtime and storage type. For PVCs and other volumes, use fsGroup, an init container, or storage-side ownership when required — see SecurityContext examples for fsGroup patterns.

Use persistent storage where needed

Ephemeral emptyDir is wrong for data that must survive Pod replacement. Move durable state to a PVC or an external service:

volumeMounts:
  - name: uploads
    mountPath: /var/lib/app/uploads
volumes:
  - name: uploads
    persistentVolumeClaim:
      claimName: app-uploads

Persistent state does not belong in the image layer. Rebuilding or retagging the image should never be required to preserve customer uploads or database files. This article does not walk through PV provisioning; treat PVC wiring as a separate storage task once you know which paths are durable.

Run as non-root

Set Pod-level identity so every container starts as an unprivileged user the image supports:

spec:
  securityContext:
    runAsNonRoot: true
    runAsUser: 101
    runAsGroup: 101
    seccompProfile:
      type: RuntimeDefault

runAsNonRoot: true instructs the kubelet to prevent the container from starting as UID 0. Setting an explicit non-zero runAsUser also makes the expected runtime identity unambiguous. Separate Pod Security Admission policies may reject an incompatible Pod specification during admission. The nginx-unprivileged image expects UID 101.

Build the image for non-root execution before you rely on Pod securityContext. A binary may be owned by root while still being executed by a non-root process. Runtime identity comes from the image's configured user and the Pod or container securityContext. Set runAsNonRoot and an explicit numeric UID when you need Kubernetes to enforce that expectation.

Reduce privileges

Add container-level hardening after the writable mounts work:

securityContext:
  allowPrivilegeEscalation: false
  readOnlyRootFilesystem: true
  capabilities:
    drop:
      - ALL

For capability add-backs, seccomp customization, and the difference between privileged and allowPrivilegeEscalation, use Linux capabilities in SecurityContext and seccomp profiles. This lab drops ALL and needs no added capability for nginx serving HTTP on port 8080.

Externalize configuration

Bake no environment-specific settings into the image when you can inject them at deploy time:

Mount a ConfigMap read-only and copy files into an emptyDir only when the application must rewrite configuration at startup. When a value changes, update the ConfigMap or Secret and roll out a replacement Pod. Do not kubectl exec into the running container to edit /etc.

Apply the hardened Deployment

Combine digest pin, read-only root, writable emptyDir mounts, non-root identity, and dropped capabilities:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-immutable
  namespace: immutable-lab
  labels:
    app.kubernetes.io/name: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: web-immutable
  template:
    metadata:
      labels:
        app: web-immutable
    spec:
      automountServiceAccountToken: false
      securityContext:
        runAsNonRoot: true
        runAsUser: 101
        runAsGroup: 101
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: nginx
          image: quay.io/nginx/nginx-unprivileged@sha256:65e3e85dbaed8ba248841d9d58a899b6197106c23cb0ff1a132b7bfe0547e4c0
          ports:
            - containerPort: 8080
              name: http
          resources:
            requests:
              cpu: 100m
              memory: 128Mi
            limits:
              cpu: 500m
              memory: 256Mi
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities:
              drop:
                - ALL
          livenessProbe:
            httpGet:
              path: /
              port: 8080
            initialDelaySeconds: 10
            periodSeconds: 10
          readinessProbe:
            httpGet:
              path: /
              port: 8080
            initialDelaySeconds: 5
            periodSeconds: 5
          volumeMounts:
            - name: tmp
              mountPath: /tmp
      volumes:
        - name: tmp
          emptyDir: {}

kubectl apply -f hardened.yaml

kubectl -n immutable-lab rollout status deployment/web-immutable --timeout=120s

deployment "web-immutable" successfully rolled out

The Pod reaches Running because every required write path now lands on a declared volume.

Verify immutability

Confirm the image layer stays read-only, only approved mounts accept writes, and the configured hardening controls are active.

Try to create a file under /etc:

kubectl -n immutable-lab exec deploy/web-immutable -- touch /etc/immutable-test

touch: /etc/immutable-test: Read-only file system
command terminated with exit code 1

Writes under /etc fail. That is the expected immutability signal.

Package managers and binary replacement fail the same way on a read-only root. There is no writable path under /usr or /bin unless you mount one deliberately, which you should not do for production workloads.

Write to an approved emptyDir mount:

kubectl -n immutable-lab exec deploy/web-immutable -- sh -c 'touch /tmp/immutable-test && ls -l /tmp/immutable-test'

-rw-r--r--    1 nginx    nginx            0 Jul 28 07:37 /tmp/immutable-test

/tmp accepts the file because it is backed by emptyDir, not the image layer.

Confirm non-root execution:

kubectl -n immutable-lab exec deploy/web-immutable -- id

uid=101(nginx) gid=101(nginx) groups=101(nginx)

Verify the effective kernel controls:

kubectl -n immutable-lab exec deploy/web-immutable -- \
  sh -c "grep -E '^(NoNewPrivs|CapEff|Seccomp):' /proc/1/status"

CapEff:	0000000000000000
NoNewPrivs:	1
Seccomp:	2

CapEff all zeros confirms no effective Linux capabilities. NoNewPrivs: 1 corresponds to privilege escalation being disabled. Seccomp: 2 means the process is running in seccomp filter mode. These controls are separately recommended by Kubernetes alongside a read-only root filesystem.

Delete the Pod and let the Deployment recreate it:

kubectl -n immutable-lab delete pod -l app=web-immutable --wait=true

kubectl -n immutable-lab rollout status deployment/web-immutable --timeout=120s

Check whether the ephemeral file survived:

kubectl -n immutable-lab exec deploy/web-immutable -- sh -c 'ls /tmp/immutable-test 2>&1'

ls: /tmp/immutable-test: No such file or directory
command terminated with exit code 1

Ephemeral data is gone after Pod replacement, which is correct for emptyDir.

Confirm the running Pod still resolves the same digest:

kubectl -n immutable-lab get pod -l app=web-immutable -o jsonpath='{.items[0].status.containerStatuses[0].imageID}{"\n"}'

quay.io/nginx/nginx-unprivileged@sha256:65e3e85dbaed8ba248841d9d58a899b6197106c23cb0ff1a132b7bfe0547e4c0

The digest is unchanged across restart. Pair this check with deployment verification in CI so production rollouts reject drift.

Debug without modification

When a read-only Pod misbehaves, gather evidence without installing packages into the production container:

Do not apt install, dnf install, or copy a static binary into the production container filesystem. That breaks immutability and hides the real mount or permission gap.

Common problems

What's Next

• Kubernetes Audit Logging with Policy Examples
• Kubernetes Runtime Security with Falco
• Kubernetes Incident Response and Forensics

References

• Kubernetes Configure a Security Context for a Pod or Container
• Kubernetes Application Security Checklist — read-only root filesystem and container hardening recommendations
• Kubernetes Images — digest-pinned image behavior
• Kubernetes Volumes
• Kubernetes Pod Security Standards
• NGINX Unprivileged Docker Image — UID, port, PID, and temporary-path behavior

Summary

You hardened a real nginx workload for runtime immutability on Kubernetes 1.36.3: digest-pinned image, readOnlyRootFilesystem, an explicit /tmp emptyDir mount, non-root UID 101, dropped capabilities, and RuntimeDefault seccomp. The read-only failure log named /tmp/proxy_temp before any volume work began, which is the pattern to repeat on your own applications — enable read-only mode, read the first write error, mount only what the process needs.

Immutability is replacement, not live patching. Configuration changes flow through ConfigMaps, Secrets, and new Pods. Ephemeral files on emptyDir disappear when the Pod is recreated, which is desirable for cache and temp data but wrong for durable uploads — those belong on a PVC or external store.

Before you enforce these settings cluster-wide, scan manifests with Kubesec and KubeLinter and align namespaces with Pod Security Standards. The Restricted Pod Security Standard requires non-root execution, disabled privilege escalation, an allowed seccomp profile, and dropped capabilities. A read-only root filesystem is additional application-hardening guidance rather than a Restricted-profile requirement; designing writable mounts early keeps remediation cheap.