Allow Trusted Images with ImagePolicyWebhook

ImagePolicyWebhook lets the API server ask an external HTTPS service whether Pod images are trusted before the Pod is stored. This walkthrough builds a small ImageReview server on a kubeadm control plane running Kubernetes 1.36.3, configures defaultAllow and TTL caching, enables the admission plugin on the static kube-apiserver manifest, and tests allowed, denied, and webhook-unavailable scenarios.

Correct failure-behavior terminology

defaultAllow: false means fail-closed: if the ImageReview service is down, image admission is denied. defaultAllow: true means fail-open: Pods may be created without a policy decision when the backend is unavailable.

Understand the admission flow

Pod create/update request
  ↓
Authentication
  ↓
Authorization (RBAC)
  ↓
ImagePolicyWebhook admission plugin
  ↓
HTTPS POST with ImageReview JSON
  ↓
Webhook returns allowed or denied
  ↓
API server admits or rejects the Pod object

ImagePolicyWebhook runs during admission, before the Pod is persisted and scheduled. The webhook receives image reference strings from containers, initContainers, and ephemeralContainers. It does not pull images from a registry or scan layers.

Admission rejection happens before a Pod exists in etcd. Denied Pods are not persisted, so they have no Pod status or container logs. The denial is returned to the client and can also appear in API-server or audit logs.

Define a simple trust policy

The lab webhook enforces a narrow policy suitable for learning:

• Allow only registry.k8s.io images
• Require an immutable digest (@sha256:…)
• Deny the :latest tag
• Deny images outside registry.k8s.io
• Return a clear reason string on denial

The code allows every repository under registry.k8s.io; it does not maintain a repository-level allow list.

Cosign verification and Trivy scanning stay outside this service. Production teams often call those tools from CI and store results as signed attestations; this webhook only enforces registry and digest rules at admission time.

Build the webhook service

Create the webhook directory on the control-plane node. The API server mounts the host path /opt/image-policy-webhook at /etc/kubernetes/image-policy-webhook inside the static Pod:

mkdir -p /opt/image-policy-webhook/certs

Save the policy server as /opt/image-policy-webhook/server.py. It uses only the Python standard library—no Flask or FastAPI dependency:

#!/usr/bin/env python3
"""Minimal ImageReview webhook for ImagePolicyWebhook admission plugin."""
import json
import logging
import re
import ssl
import sys
from http.server import BaseHTTPRequestHandler, HTTPServer

PORT = 8443
CERT = "/opt/image-policy-webhook/certs/tls.crt"
KEY = "/opt/image-policy-webhook/certs/tls.key"

SHA256_DIGEST = re.compile(r"^[0-9a-f]{64}$")

logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s %(levelname)s %(message)s",
    handlers=[
        logging.StreamHandler(sys.stdout),
        logging.FileHandler("/var/log/image-policy-webhook.log"),
    ],
)
log = logging.getLogger("image-policy-webhook")


def evaluate_image(image: str) -> tuple[bool, str]:
    if not image:
        return False, "denied: empty image reference"

    if not image.startswith("registry.k8s.io/"):
        return False, f"denied: only registry.k8s.io is allowed: {image}"

    image_name = image.split("@", 1)[0]
    if image_name.endswith(":latest"):
        return False, f"denied: :latest is not allowed: {image}"

    _, separator, digest = image.rpartition("@sha256:")
    if not separator or not SHA256_DIGEST.fullmatch(digest):
        return False, f"denied: image must use a complete @sha256 digest: {image}"

    return True, ""


def evaluate_review(spec: dict) -> tuple[bool, str, list[str]]:
    containers = spec.get("containers") or []
    images = [container.get("image", "") for container in containers]

    if not images:
        return False, "denied: ImageReview contains no images", images

    for image in images:
        allowed, reason = evaluate_image(image)
        if not allowed:
            return False, reason, images

    return True, "", images


class Handler(BaseHTTPRequestHandler):
    def log_message(self, fmt, *args):
        log.info("%s - %s", self.address_string(), fmt % args)

    def do_GET(self):
        if self.path in ("/healthz", "/healthz/"):
            self.send_response(200)
            self.send_header("Content-Type", "text/plain")
            self.end_headers()
            self.wfile.write(b"ok\n")
            return
        self.send_response(404)
        self.end_headers()

    def do_POST(self):
        length = int(self.headers.get("Content-Length", 0))
        body = self.rfile.read(length)
        log.info("ImageReview request: %s", body.decode("utf-8", errors="replace")[:2000])
        review = json.loads(body)
        spec = review.get("spec") or {}
        allowed, reason, images = evaluate_review(spec)
        response = {
            "apiVersion": review.get("apiVersion", "imagepolicy.k8s.io/v1alpha1"),
            "kind": review.get("kind", "ImageReview"),
            "status": {"allowed": allowed, "reason": reason},
        }
        log.info(
            "ImageReview response: allowed=%s reason=%s images=%s",
            allowed,
            reason,
            images,
        )
        out = json.dumps(response).encode()
        self.send_response(200)
        self.send_header("Content-Type", "application/json")
        self.end_headers()
        self.wfile.write(out)


def main():
    httpd = HTTPServer(("127.0.0.1", PORT), Handler)
    ctx = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER)
    ctx.load_cert_chain(CERT, KEY)
    httpd.socket = ctx.wrap_socket(httpd.socket, server_side=True)
    log.info("Image policy webhook listening on https://127.0.0.1:%s", PORT)
    httpd.serve_forever()


if __name__ == "__main__":
    main()

Generate a TLS certificate signed by the cluster CA. The API server must trust the CA that signed the webhook certificate. This lab signs the certificate with the kubeadm cluster CA and embeds that CA in the webhook kubeconfig. A separate self-signed certificate can also work when that certificate is explicitly configured as the trusted CA.

cd /opt/image-policy-webhook/certs
openssl req -newkey rsa:2048 -keyout tls.key -out tls.csr -nodes \
  -subj "/CN=image-policy-webhook" \
  -addext "subjectAltName=DNS:localhost,IP:127.0.0.1,IP:192.168.56.108"
openssl x509 -req -in tls.csr \
  -CA /etc/kubernetes/pki/ca.crt -CAkey /etc/kubernetes/pki/ca.key \
  -CAcreateserial -out tls.crt -days 365 \
  -extensions v3_req \
  -extfile <(printf "[v3_req]\nsubjectAltName=DNS:localhost,IP:127.0.0.1,IP:192.168.56.108\n")

chmod 600 /opt/image-policy-webhook/certs/tls.key

Install a systemd unit so the webhook starts on boot:

cat >/etc/systemd/system/image-policy-webhook.service <<'EOF'
[Unit]
Description=Kubernetes ImagePolicyWebhook backend
After=network-online.target
Wants=network-online.target

[Service]
ExecStart=/usr/bin/python3 /opt/image-policy-webhook/server.py
Restart=on-failure
RestartSec=3

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl enable --now image-policy-webhook

Confirm the health endpoint responds over HTTPS from the control-plane host:

curl --cacert /etc/kubernetes/pki/ca.crt https://127.0.0.1:8443/healthz

ok

The API server static Pod calls the webhook at https://127.0.0.1:8443 from inside the control-plane node network namespace.

Configure webhook TLS and kubeconfig

The API server reads a kubeconfig file that points at the webhook HTTPS endpoint and trusts the CA certificate.

Create /opt/image-policy-webhook/kubeconfig with the cluster CA the API server already trusts:

CA_DATA=$(base64 -w0 /etc/kubernetes/pki/ca.crt)

cat >/opt/image-policy-webhook/kubeconfig <<EOF
apiVersion: v1
kind: Config
clusters:
- cluster:
    certificate-authority-data: ${CA_DATA}
    server: https://127.0.0.1:8443
  name: image-policy-webhook
contexts:
- context:
    cluster: image-policy-webhook
  name: default
current-context: default
users: []
EOF

Verify that the placeholder is gone:

grep certificate-authority-data /opt/image-policy-webhook/kubeconfig

certificate-authority-data must contain base64-encoded PEM data.

Paths under /etc/kubernetes/image-policy-webhook/ are valid inside the API server container because the host directory is bind-mounted there.

Test reachability from the control-plane host before you edit the API server:

curl --cacert /etc/kubernetes/pki/ca.crt https://127.0.0.1:8443/healthz

ok

The health check proves only that HTTPS responds. Before you modify the API server, send an ImageReview that contains one trusted and one untrusted image:

curl --cacert /etc/kubernetes/pki/ca.crt \
  -H 'Content-Type: application/json' \
  --data-binary @- \
  https://127.0.0.1:8443/ <<'EOF'
{
  "apiVersion": "imagepolicy.k8s.io/v1alpha1",
  "kind": "ImageReview",
  "spec": {
    "namespace": "image-policy-test",
    "containers": [
      {
        "image": "registry.k8s.io/pause@sha256:278fb9dbcca9518083ad1e11276933a2e96f23de604a3a08cc3c80002767d24c"
      },
      {
        "image": "registry.k8s.io/pause:3.10"
      }
    ]
  }
}
EOF

{"apiVersion": "imagepolicy.k8s.io/v1alpha1", "kind": "ImageReview", "status": {"allowed": false, "reason": "denied: image must use a complete @sha256 digest: registry.k8s.io/pause:3.10"}}

status.allowed: false on the second image proves the webhook evaluates every entry in spec.containers[], not only the first container.

Client certificate authentication is optional for this lab. Production deployments often add mutual TLS between the API server and the policy service.

Create ImagePolicyWebhook configuration

This lab keeps the ImagePolicyWebhook settings in a separate plugin-config.yaml and references it through plugins[].path. Upstream also supports inline plugins[].configuration.imagePolicy; an unknown field "imagePolicy" error usually means the file was malformed or decoded as the wrong configuration type, not that inline configuration is unsupported generally.

Create /opt/image-policy-webhook/admission-config.yaml:

apiVersion: apiserver.config.k8s.io/v1
kind: AdmissionConfiguration
plugins:
- name: ImagePolicyWebhook
  path: /etc/kubernetes/image-policy-webhook/plugin-config.yaml

Create /opt/image-policy-webhook/plugin-config.yaml:

imagePolicy:
  kubeConfigFile: /etc/kubernetes/image-policy-webhook/kubeconfig
  allowTTL: 30
  denyTTL: 30
  retryBackoff: 500
  defaultAllow: false

Field meanings:

Start with defaultAllow: false in production-minded labs. Changing defaultAllow requires editing plugin-config.yaml and restarting the API server—the admission files are read at apiserver start, not hot-reloaded.

Enable the admission plugin

Back up the static Pod manifest outside the watched manifests directory, then add the ImagePolicyWebhook plugin and admission configuration file.

install -d -m 700 /etc/kubernetes/backup

cp -a /etc/kubernetes/manifests/kube-apiserver.yaml \
  "/etc/kubernetes/backup/kube-apiserver.yaml.$(date +%F-%H%M%S)"

Add or extend these flags in the kube-apiserver container command array:

--enable-admission-plugins=NodeRestriction,ImagePolicyWebhook
--admission-control-config-file=/etc/kubernetes/image-policy-webhook/admission-config.yaml
--runtime-config=imagepolicy.k8s.io/v1alpha1=true

Upstream documentation still instructs administrators to enable the alpha ImageReview API group. This lab cluster appeared to operate without the flag during testing, but include it for a reproducible workflow.

ImagePolicyWebhook does not control the actual kubeadm static Pods because the kubelet starts them directly without API-server admission. It can reject their mirror Pod objects, while the underlying static Pods continue running. With this digest-only policy, mirror objects for the API server, scheduler, controller manager, or etcd may fail to register unless their image references also satisfy the policy.

Static Pods explicitly bypass API-server admission; only their mirror objects are submitted to the API.

Add a volume mount and hostPath volume so the API server container can read webhook files:

volumeMounts:
    - mountPath: /etc/kubernetes/image-policy-webhook
      name: image-policy-webhook
      readOnly: true
    volumes:
    - hostPath:
        path: /opt/image-policy-webhook
        type: Directory
      name: image-policy-webhook

The kubelet recreates the API server Pod when the manifest changes. Wait until the API server returns a successful /readyz response:

crictl ps --name kube-apiserver

CONTAINER           IMAGE               CREATED             STATE               NAME                ATTEMPT             POD ID              POD                     NAMESPACE
dd97d10404b81       3400718b4dd57       2 minutes ago       Running             kube-apiserver      1                   6e99995046d89       kube-apiserver-k8s-cp   kube-system

until kubectl get --raw=/readyz >/dev/null 2>&1; do
  sleep 2
done

kubectl get --raw=/readyz
kubectl get nodes

ok
NAME       STATUS   ROLES           AGE   VERSION
k8s-cp     Ready    control-plane   19h   v1.36.3
worker01   Ready    <none>          19h   v1.36.3

The API-server container can be healthy in crictl even when its mirror Pod is missing from kubectl get pods -n kube-system. Static Pods run under kubelet supervision; mirror creation is a separate API admission step that this digest-only policy can reject.

Confirm kubectl get nodes still succeeds before you run Pod admission tests.

Test an allowed Pod

Create a test namespace and a Pod that satisfies the lab policy—registry.k8s.io image with a pinned digest:

kubectl create namespace image-policy-test

kubectl apply -f - <<'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: ipw-test-a
  namespace: image-policy-test
spec:
  containers:
  - name: pause
    image: registry.k8s.io/pause@sha256:278fb9dbcca9518083ad1e11276933a2e96f23de604a3a08cc3c80002767d24c
EOF

pod/ipw-test-a created

Admission succeeded. Confirm the Pod object exists separately from image pull:

kubectl get pod ipw-test-a -o wide

NAME         READY   STATUS    RESTARTS   AGE   IP       NODE       NOMINATED NODE   READINESS GATES
ipw-test-a   0/1     Pending   0          12s   <none>   worker01   <none>           <none>

A Pending Pod still means admission passed. Image pull success depends on node connectivity to registry.k8s.io. Use the repository manifest digest from crictl inspecti or skopeo inspect, not a local image layer ID—a wrong digest such as sha256:873ed751… can pass admission but fail with ImagePullBackOff.

Test denied Pods

Mutable tag without digest

kubectl apply -f - <<'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: tag-only-pause
  namespace: image-policy-test
spec:
  containers:
  - name: pause
    image: registry.k8s.io/pause:3.10
EOF

Error from server (Forbidden): error when creating "STDIN": pods "tag-only-pause" is forbidden: image policy webhook backend denied one or more images: denied: image must use a complete @sha256 digest: registry.k8s.io/pause:3.10

Unapproved registry

kubectl apply -f - <<'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: dockerhub-nginx
  namespace: image-policy-test
spec:
  containers:
  - name: nginx
    image: docker.io/nginx:latest
EOF

Error from server (Forbidden): error when creating "STDIN": pods "dockerhub-nginx" is forbidden: image policy webhook backend denied one or more images: denied: only registry.k8s.io is allowed: docker.io/nginx:latest

The registry allow list is checked before digest and :latest rules, so foreign images are denied with the registry reason first.

Explicit :latest tag

kubectl apply -f - <<'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: latest-pause
  namespace: image-policy-test
spec:
  containers:
  - name: pause
    image: registry.k8s.io/pause:latest
EOF

Error from server (Forbidden): error when creating "STDIN": pods "latest-pause" is forbidden: image policy webhook backend denied one or more images: denied: :latest is not allowed: registry.k8s.io/pause:latest

Denied Pods never reach the API store, so kubectl logs returns NotFound and no workload logs exist.

kubectl get pod tag-only-pause -n image-policy-test

Error from server (NotFound): pods "tag-only-pause" not found

Test webhook unavailability

Stop the webhook service and attempt another allowed Pod with defaultAllow: false. Add a unique forwarded annotation so the result is not served from the allowTTL: 30 cache:

systemctl stop image-policy-webhook

kubectl apply -f - <<'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: ipw-test-fail-closed
  namespace: image-policy-test
  annotations:
    lab.image-policy.k8s.io/cache-buster: "fail-closed-1"
spec:
  containers:
  - name: pause
    image: registry.k8s.io/pause@sha256:278fb9dbcca9518083ad1e11276933a2e96f23de604a3a08cc3c80002767d24c
EOF

Error from server (Forbidden): error when creating "STDIN": pods "ipw-test-fail-closed" is forbidden: error while calling image policy webhook: Post "https://127.0.0.1:8443/?timeout=30s": dial tcp 127.0.0.1:8443: connect: connection refused

With defaultAllow: false, the API server fails closed and rejects the Pod.

Edit /opt/image-policy-webhook/plugin-config.yaml to set defaultAllow: true, then restart the API server so it reloads admission configuration. plugin-config.yaml is read at API-server startup; touching an unchanged static-Pod manifest does not provide a reliable content change:

APISERVER_ID=$(
  crictl ps --name kube-apiserver -q | head -n 1
)

crictl stop "$APISERVER_ID"

until kubectl get --raw=/readyz >/dev/null 2>&1; do
  sleep 2
done

Wait until the API server returns a successful /readyz response, then re-run the Pod manifest with a different cache-buster value. The webhook is still stopped from the fail-closed test:

kubectl apply -f - <<'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: ipw-test-fail-open
  namespace: image-policy-test
  annotations:
    lab.image-policy.k8s.io/cache-buster: "fail-open-1"
spec:
  containers:
  - name: pause
    image: registry.k8s.io/pause@sha256:278fb9dbcca9518083ad1e11276933a2e96f23de604a3a08cc3c80002767d24c
EOF

pod/ipw-test-fail-open created

Fail-open behavior keeps Pod admission available when the policy service is down, but Pods may be admitted without an image-policy decision. The scheduler acts after admission; defaultAllow controls the admission decision when the webhook call fails. Restore production posture after the test:

systemctl start image-policy-webhook

Set defaultAllow back to false in plugin-config.yaml and stop the API server container with the same crictl stop method so the kubelet recreates it with the restored settings.

ImagePolicyWebhook alternatives

ImagePolicyWebhook is a built-in plugin with a fixed ImageReview contract. Many teams adopt one of these instead or in addition:

This article implements ImagePolicyWebhook only. Link to Cosign for signature verification rather than reimplementing it in Python.

Inspect verification output

When debugging, tail webhook logs:

journalctl -u image-policy-webhook -f

ImageReview request: {"apiVersion":"imagepolicy.k8s.io/v1alpha1","kind":"ImageReview","spec":{"containers":[{"image":"registry.k8s.io/pause:3.10"}]}}
ImageReview response: allowed=False reason=denied: image must use a complete @sha256 digest: registry.k8s.io/pause:3.10 images=['registry.k8s.io/pause:3.10']

Each log line corresponds to an API server ImageReview POST. Pair logs with kubectl errors to see which image reference triggered a deny reason.

ImageReview responses include:

• status.allowed — boolean admission decision
• status.reason — human-readable deny text returned to the API server
• Request spec.containers[].image — evaluated references
• Optional spec.annotations matching *.image-policy.k8s.io/* for break-glass workflows

Troubleshooting

What's Next

• Scan Kubernetes YAML with Kubesec and KubeLinter
• Secure Container Image Promotion in CI/CD
• kubectl logs, Events and describe with Examples

References

• Kubernetes admission controllers — ImagePolicyWebhook
• Image Policy API (v1alpha1)
• Admission configuration

Summary

ImagePolicyWebhook sends ImageReview requests to your HTTPS backend before Pods are admitted. Configure defaultAllow in plugin-config.yaml, TTL caching, and a kubeconfig backed by the cluster CA, then enable the plugin on the static kube-apiserver manifest.

The lab allowed registry.k8s.io/pause@sha256:278fb9dbcca9518083ad1e11276933a2e96f23de604a3a08cc3c80002767d24c, denied mutable tags and foreign registries with explicit reasons, and demonstrated fail-closed behavior when the webhook stopped with defaultAllow: false. Pair digest and registry rules with Cosign verification and Trivy scanning in your supply-chain pipeline.