Kubernetes Audit Logging with Policy Examples

Kubernetes audit logs answer API accountability questions: who changed RBAC, who read a Secret, who ran kubectl exec, and which requests failed authorization. This walkthrough enables a file-backed audit log on a kubeadm control plane running 1.36.3, applies a focused policy, generates controlled API activity in namespace audit-lab, and filters /var/log/kubernetes/audit/audit.log with jq.

What audit logs record

Audit events describe Kubernetes API requests, not processes inside containers. Each line in audit.log is a JSON object that helps you reconstruct:

• What happened (verb, requestURI, objectRef)
• When (requestReceivedTimestamp, stage)
• Who initiated it (user.username, user.groups, impersonatedUser)
• Which resource (objectRef.apiGroup, resource, namespace, name)
• From where (sourceIPs, userAgent)
• The result (responseStatus.code, responseStatus.status, and responseStatus.message)

Audit logging complements RBAC design reviews. RBAC defines what is allowed; audit logs show what actually occurred after deployment.

Understand audit stages

The API server can emit one or more audit records per request at different stages:

This guide omits RequestReceived at policy scope so each normal request produces one final record. Long-running requests such as exec, port forwarding, and watches may finish at ResponseStarted instead.

Understand audit levels

Higher levels help debugging but increase volume and sensitivity. RequestResponse on Secrets, ConfigMaps, or TokenReview objects can persist credentials in audit.log. Default sensitive resources to Metadata unless a compliance control explicitly requires bodies—and then restrict log access accordingly.

Design an audit policy

Save the policy on the control-plane node at /etc/kubernetes/audit-policy.yaml. Rules are evaluated top to bottom; the first match wins.

apiVersion: audit.k8s.io/v1
kind: Policy
omitStages:
  - RequestReceived
rules:
  - level: None
    users:
      - system:kube-scheduler
      - system:kube-controller-manager
      - system:kube-proxy
    verbs: ["get", "list", "watch"]
    resources:
      - group: ""
        resources: ["endpoints", "services", "nodes", "pods"]

  - level: Metadata
    verbs: ["create"]
    resources:
      - group: "authentication.k8s.io"
        resources: ["tokenreviews"]
      - group: "authorization.k8s.io"
        resources:
          - subjectaccessreviews
          - selfsubjectaccessreviews
          - localsubjectaccessreviews
          - selfsubjectrulesreviews

  - level: Request
    verbs: ["create", "update", "patch", "delete"]
    resources:
      - group: "rbac.authorization.k8s.io"
        resources: ["roles", "rolebindings", "clusterroles", "clusterrolebindings"]

  - level: Metadata
    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
    resources:
      - group: ""
        resources: ["secrets"]

  - level: Request
    verbs: ["create", "delete", "deletecollection"]
    resources:
      - group: ""
        resources: ["pods"]

  - level: Request
    verbs: ["get", "create"]
    resources:
      - group: ""
        resources: ["pods/exec", "pods/portforward"]

  - level: Request
    verbs: ["create", "update", "patch", "delete"]
    resources:
      - group: ""
        resources: ["nodes"]

  - level: Metadata
    verbs: ["create"]
    resources:
      - group: ""
        resources: ["serviceaccounts/token"]

  - level: None
    users: ["system:apiserver"]
    verbs: ["get", "list", "watch"]

  - level: Metadata

This policy omits RequestReceived globally so normal requests are investigated using their final ResponseComplete result. Long-running requests such as exec, port forwarding, and watches may instead produce ResponseStarted. Kubernetes can generate an event at each request stage, and policy-level and rule-level omitStages values are combined.

This policy:

• Drops high-volume list/watch noise from core controllers
• Records authentication and authorization reviews at Metadata (TokenReview and SubjectAccessReview resources, including kubectl auth can-i checks)
• Captures RBAC changes at Request (who bound which Role)
• Logs Secret touch events at Metadata without bodies
• Records Pod create/delete and exec / portforward subresources at Request (get for WebSocket kubectl exec on Kubernetes 1.31+, create for older POST-based clients)
• Records node changes and ServiceAccount token creation
• Ends with a Metadata catch-all for remaining API traffic

Request level stores the submitted API request object. For Pod creation, that can include literal environment values, commands, arguments, annotations, and other Pod-spec content. Do not place plaintext credentials directly in Pod manifests.

For pods/exec, audit logging records the exec API request and its initial options. It does not record the terminal byte stream or command output. If a user starts an interactive shell, commands typed inside that shell are not individually captured by the API audit event. The audit API defines requestObject as the submitted object at Request level; it does not capture subsequent container activity.

Place specific rules before broad ones. If Pod exec events are missing, a None rule above the pods/exec entry is the first place to check.

Configure the API server

On the control-plane node, install jq and confirm crictl before you edit the static Pod manifest:

# RHEL, Rocky Linux, AlmaLinux, Fedora
dnf install -y jq

# Ubuntu and Debian
apt-get update && apt-get install -y jq

jq --version

jq-1.7.1

crictl --version

crictl version v1.36.0

Create the log directory and save the policy file:

install -d -m 700 /var/log/kubernetes/audit

cat > /etc/kubernetes/audit-policy.yaml <<'EOF'
apiVersion: audit.k8s.io/v1
kind: Policy
omitStages:
  - RequestReceived
rules:
  - level: None
    users:
      - system:kube-scheduler
      - system:kube-controller-manager
      - system:kube-proxy
    verbs: ["get", "list", "watch"]
    resources:
      - group: ""
        resources: ["endpoints", "services", "nodes", "pods"]

  - level: Metadata
    verbs: ["create"]
    resources:
      - group: "authentication.k8s.io"
        resources: ["tokenreviews"]
      - group: "authorization.k8s.io"
        resources:
          - subjectaccessreviews
          - selfsubjectaccessreviews
          - localsubjectaccessreviews
          - selfsubjectrulesreviews

  - level: Request
    verbs: ["create", "update", "patch", "delete"]
    resources:
      - group: "rbac.authorization.k8s.io"
        resources: ["roles", "rolebindings", "clusterroles", "clusterrolebindings"]

  - level: Metadata
    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
    resources:
      - group: ""
        resources: ["secrets"]

  - level: Request
    verbs: ["create", "delete", "deletecollection"]
    resources:
      - group: ""
        resources: ["pods"]

  - level: Request
    verbs: ["get", "create"]
    resources:
      - group: ""
        resources: ["pods/exec", "pods/portforward"]

  - level: Request
    verbs: ["create", "update", "patch", "delete"]
    resources:
      - group: ""
        resources: ["nodes"]

  - level: Metadata
    verbs: ["create"]
    resources:
      - group: ""
        resources: ["serviceaccounts/token"]

  - level: None
    users: ["system:apiserver"]
    verbs: ["get", "list", "watch"]

  - level: Metadata
EOF

Back up the current static Pod manifest before you edit it. Keep the backup outside /etc/kubernetes/manifests/. The kubelet processes every non-hidden file in the static-Pod directory, regardless of extension. Kubernetes explicitly warns that non-hidden backup files in this directory can produce undefined behavior.

cp -a /etc/kubernetes/manifests/kube-apiserver.yaml \
  /root/kube-apiserver.yaml.audit-baseline-$(date +%Y%m%d-%H%M%S)

OLD_APISERVER_ID=$(crictl ps -q --name kube-apiserver | head -1)
echo "Old container: $OLD_APISERVER_ID"

Edit /etc/kubernetes/manifests/kube-apiserver.yaml. Add audit flags to the kube-apiserver command array (after --allow-privileged=true is a readable anchor):

- --audit-policy-file=/etc/kubernetes/audit-policy.yaml
    - --audit-log-path=/var/log/kubernetes/audit/audit.log
    - --audit-log-maxage=30
    - --audit-log-maxbackup=10
    - --audit-log-maxsize=100

Add volume mounts to the kube-apiserver container. Append the following entries to the existing volumeMounts list under the kube-apiserver container. Do not create a second volumeMounts: key.

- mountPath: /etc/kubernetes/audit-policy.yaml
      name: audit-policy
      readOnly: true
    - mountPath: /var/log/kubernetes/audit
      name: audit-log

Append these entries to the existing Pod-level volumes list. Do not create a second volumes: key.

- name: audit-policy
    hostPath:
      path: /etc/kubernetes/audit-policy.yaml
      type: File
  - name: audit-log
    hostPath:
      path: /var/log/kubernetes/audit
      type: DirectoryOrCreate

Keep kube-apiserver as the first element of the command array. If you use ImagePolicyWebhook or other admission flags, preserve them when you add audit settings.

The kubelet watches /etc/kubernetes/manifests/ and recreates the static Pod when the file changes. Keep only active manifest files in that directory—stale copies can prevent the kubelet from applying your intended spec.

Verify API server recovery

The kubelet scans /etc/kubernetes/manifests/ periodically. An immediate /readyz check can still reach the old API-server container. Wait for container recreation, then confirm readiness and audit logging:

NEW_APISERVER_ID=""

for attempt in $(seq 1 60); do
  NEW_APISERVER_ID=$(crictl ps -q --name kube-apiserver | head -1)

  if [ -n "$NEW_APISERVER_ID" ] &&
     [ "$NEW_APISERVER_ID" != "$OLD_APISERVER_ID" ]; then
    echo "New container: $NEW_APISERVER_ID"
    break
  fi

  sleep 2
done

if [ -z "$NEW_APISERVER_ID" ] ||
   [ "$NEW_APISERVER_ID" = "$OLD_APISERVER_ID" ]; then
  echo "API server was not recreated" >&2
  crictl ps -a --name kube-apiserver
  exit 1
fi

export KUBECONFIG=/etc/kubernetes/admin.conf
APISERVER_READY=false

for attempt in $(seq 1 60); do
  if kubectl --request-timeout=5s get --raw='/readyz' \
    >/dev/null 2>&1; then
    APISERVER_READY=true
    echo "Replacement API server is ready"
    break
  fi

  sleep 2
done

if [ "$APISERVER_READY" != true ]; then
  echo "Replacement API server did not become ready" >&2
  crictl ps -a --name kube-apiserver
  crictl logs "$NEW_APISERVER_ID"
  exit 1
fi

kubectl get --raw='/readyz?verbose'
kubectl get nodes

[+]ping ok
[+]log ok
[+]etcd ok
[+]etcd-readiness ok

After the API server starts, verify audit-directory ownership and permissions:

stat -c '%a %U:%G %n' \
  /var/log/kubernetes/audit \
  /var/log/kubernetes/audit/audit.log

700 root:root /var/log/kubernetes/audit
600 root:root /var/log/kubernetes/audit/audit.log

The Kubernetes 1.36 audit backend creates its directory with mode 0700 and its log file with mode 0600; precreating the directory as 0755 leaves it unnecessarily accessible.

Verify the audit flags on the new process and confirm the log file grows after a test request:

tr '\0' '\n' \
  < /proc/$(pgrep -xo kube-apiserver)/cmdline |
  grep '^--audit-'

--audit-log-maxage=30
--audit-log-maxbackup=10
--audit-log-maxsize=100
--audit-log-path=/var/log/kubernetes/audit/audit.log
--audit-policy-file=/etc/kubernetes/audit-policy.yaml

BEFORE=$(stat -c %s /var/log/kubernetes/audit/audit.log)
kubectl get namespace default >/dev/null
AFTER=$(stat -c %s /var/log/kubernetes/audit/audit.log)

printf 'Audit log bytes: %s -> %s\n' "$BEFORE" "$AFTER"
test "$AFTER" -gt "$BEFORE"

Audit log bytes: 2457600 -> 2458112

Both the policy file and log directory must be mounted into the API-server Pod, as shown in the manifest edits above. If /readyz fails, the log file stays empty, or the byte count does not increase, inspect the latest kube-apiserver container log through crictl logs before you proceed.

Generate test activity

Create an isolated namespace for audit demos:

kubectl create namespace audit-lab

kubectl get namespace audit-lab -o name

Create, read, and delete a ConfigMap:

kubectl -n audit-lab create configmap audit-demo \
  --from-literal=key=value

kubectl -n audit-lab get configmap audit-demo -o name

kubectl -n audit-lab delete configmap audit-demo

Create a Secret and read it:

kubectl -n audit-lab create secret generic audit-secret --from-literal=token=demo-token-12345

kubectl -n audit-lab get secret audit-secret -o name

Create RBAC objects:

kubectl -n audit-lab create role audit-reader2 --verb=get --resource=secrets

kubectl -n audit-lab create rolebinding audit-reader2-binding --role=audit-reader2 --user=[email protected]

Trigger a forbidden delete with impersonation:

kubectl --as=[email protected] -n audit-lab delete pod forbidden-test --ignore-not-found

Error from server (Forbidden): pods "forbidden-test" is forbidden: User "[email protected]" cannot delete resource "pods" in API group "" in the namespace "audit-lab"

Run kubectl exec against a test Pod:

kubectl -n audit-lab run exec-demo2 --image=registry.k8s.io/e2e-test-images/agnhost:2.45 --restart=Never --command -- sleep 3600

kubectl -n audit-lab wait --for=condition=Ready pod/exec-demo2 --timeout=120s

kubectl -n audit-lab exec exec-demo2 -- /bin/true

Request a ServiceAccount token:

kubectl -n audit-lab create serviceaccount audit-sa

kubectl -n audit-lab create token audit-sa \
  --duration=10m >/dev/null

Each command above produces API traffic the policy should record.

Attribute activity

Focus on these JSON fields when you investigate an event:

When kubectl --as is used, the audit record keeps the real admin in user.username and places the effective identity in impersonatedUser:

{
  "user": { "username": "kubernetes-admin" },
  "impersonatedUser": { "username": "[email protected]" },
  "verb": "delete",
  "responseStatus": { "code": 403 }
}

Use impersonatedUser for authorization forensics when admins impersonate CI users or break-glass accounts.

Filter events with jq

Each line in audit.log is one JSON object. Audit resource and subresource names are matched separately; for example, pods does not mean all Pod subresources, while pods/exec specifically matches exec.

Failed requests (responseStatus.code == 403):

jq -c 'select(.responseStatus.code == 403) | {user:.user.username, impersonated:.impersonatedUser.username, verb:.verb, uri:.requestURI, status:.responseStatus.code}' /var/log/kubernetes/audit/audit.log | tail -1

{"user":"kubernetes-admin","impersonated":"[email protected]","verb":"delete","uri":"/api/v1/namespaces/audit-lab/pods/forbidden-test","status":403}

Secret reads (metadata only):

jq -c 'select(.objectRef.resource == "secrets" and .objectRef.name == "audit-secret") | {user:.user.username, verb:.verb, ns:.objectRef.namespace, name:.objectRef.name, level:.level}' /var/log/kubernetes/audit/audit.log | tail -1

{"user":"kubernetes-admin","verb":"get","ns":"audit-lab","name":"audit-secret","level":"Metadata"}

RBAC changes:

jq -c 'select(.objectRef.resource == "rolebindings" and .objectRef.name == "audit-reader2-binding") | {user:.user.username, verb:.verb, resource:.objectRef.resource, name:.objectRef.name}' /var/log/kubernetes/audit/audit.log | tail -1

{"user":"kubernetes-admin","verb":"create","resource":"rolebindings","name":"audit-reader2-binding"}

Pod exec (pods/exec appears in requestURI):

jq -c 'select(.requestURI | contains("pods/exec")) | {user:.user.username, verb:.verb, uri:.requestURI}' /var/log/kubernetes/audit/audit.log | tail -1

{"user":"kubernetes-admin","verb":"get","uri":"/api/v1/namespaces/audit-lab/pods/exec-demo2/exec?command=%2Fbin%2Ftrue&container=exec-demo2&stderr=true&stdout=true"}

Activity inside audit-lab (not the cluster-scoped Namespace object):

jq -c '
  select(
    .objectRef.namespace == "audit-lab" and
    (
      .stage == "ResponseComplete" or
      .stage == "ResponseStarted"
    )
  )
  | {
      user: .user.username,
      impersonated: .impersonatedUser.username,
      verb: .verb,
      resource: .objectRef.resource,
      subresource: .objectRef.subresource,
      name: .objectRef.name,
      status: .responseStatus.code
    }
' /var/log/kubernetes/audit/audit.log | tail -5

{"user":"kubernetes-admin","impersonated":null,"verb":"get","resource":"pods","subresource":null,"name":"exec-demo2","status":200}
{"user":"kubernetes-admin","impersonated":null,"verb":"get","resource":"pods","subresource":"exec","name":"exec-demo2","status":101}
{"user":"kubernetes-admin","impersonated":null,"verb":"get","resource":"pods","subresource":"exec","name":"exec-demo2","status":101}
{"user":"kubernetes-admin","impersonated":null,"verb":"create","resource":"serviceaccounts","subresource":null,"name":"audit-sa","status":201}
{"user":"kubernetes-admin","impersonated":null,"verb":"create","resource":"serviceaccounts","subresource":"token","name":"audit-sa","status":201}

The last lines from this lab run include a Pod read, pods/exec subresource events (status 101 for the WebSocket upgrade), ServiceAccount creation, and a token request—all from the same namespace filter.

ServiceAccount token creation:

jq -c 'select(.requestURI | contains("serviceaccounts/audit-sa/token")) | {user:.user.username, verb:.verb, uri:.requestURI, level:.level}' /var/log/kubernetes/audit/audit.log | tail -1

{"user":"kubernetes-admin","verb":"create","uri":"/api/v1/namespaces/audit-lab/serviceaccounts/audit-sa/token","level":"Metadata"}

Source IP and user agent for a ConfigMap read:

jq -c '
  select(
    .objectRef.resource == "configmaps" and
    .objectRef.namespace == "audit-lab" and
    .objectRef.name == "audit-demo" and
    .verb == "get" and
    .stage == "ResponseComplete"
  )
  | {
      user: .user.username,
      sourceIPs: .sourceIPs,
      userAgent: .userAgent,
      verb: .verb
    }
' /var/log/kubernetes/audit/audit.log | tail -1

{"user":"kubernetes-admin","sourceIPs":["192.168.56.108"],"userAgent":"kubectl/v1.36.3 (linux/amd64) kubernetes/0f29094","verb":"get"}

Rotate and export logs

The flags in this lab configure local file rotation:

Monitor disk use on the control-plane node—audit volume grows quickly when catch-all rules log every list call. Ship rotated files to centralized storage with restricted access (object storage, syslog, or a log agent). Sync clocks across control-plane nodes with NTP so cross-node timelines align during incidents.

Treat exported audit logs like security data: encrypt in transit, limit read access, and define retention outside this article’s scope.

Protect sensitive audit content

• Prefer Metadata for Secrets, TokenReview, and ServiceAccount token requests
• Avoid RequestResponse on resources that carry credentials unless required
• Restrict /var/log/kubernetes/audit/ to root (chmod 600 on audit.log is typical)
• Never tail audit logs into public tickets or chat systems without redaction

If you must capture request bodies for debugging, use a short-lived policy in a non-production cluster and revert to metadata-only rules afterward.

Troubleshooting

What's Next

• Kubernetes Runtime Security with Falco
• Kubernetes Incident Response and Forensics

References

• Kubernetes Auditing
• Audit policy reference
• kube-apiserver command-line reference

Summary

You enabled file-backed Kubernetes audit logging on a kubeadm control plane by adding audit flags and volume mounts to the kube-apiserver static Pod, applying an ordered policy that emphasizes RBAC changes, Secret metadata, Pod lifecycle, exec, and token creation while dropping controller noise. After the replacement API server passed /readyz and audit.log grew on a test request, controlled kubectl activity in audit-lab produced JSON lines you filtered with jq to attribute ConfigMap reads, Secret reads, RoleBinding creation, impersonated forbidden deletes, and exec calls to kubernetes-admin and [email protected].

Audit logs record API intent and outcome—they do not replace runtime threat detection or workload log pipelines. Keep Secret and token bodies out of logs unless policy explicitly requires them, rotate and ship files with tight access control, and pair this visibility with RBAC reviews and kube-bench checks that flag missing audit policies on hardened clusters.