Kubernetes Runtime Security with Falco

Falco evaluates runtime events against rules and emits alerts when a process, file access, or container action looks suspicious. This walkthrough deploys Falco 0.44.1 on a kubeadm lab running Kubernetes 1.36.3, confirms syscall collection with the modern eBPF driver, triggers four controlled detections, and adds one custom rule for shell activity in a protected namespace.

What Falco detects

Falco matches events from configured sources against YAML rules. Each rule has a condition, output format, priority, and optional tags.

Common event sources include:

• Linux syscalls — execve, open, connect, and related kernel events collected through the Falco driver (kernel module or modern eBPF)
• Kubernetes audit plugin events — API audit events delivered to the k8saudit plugin, commonly by configuring the Kubernetes API server to send audit events to the plugin's webhook endpoint
• Cloud or application plugins — optional plugins for AWS CloudTrail, Okta, and other streams

Rules apply to one event source. A syscall rule does not automatically combine with a k8saudit rule into a single correlated incident. The k8saudit plugin receives Kubernetes audit events through its configured web server; it is not simply pointed at an audit-log file path. If you need cross-source timelines, forward both streams to a log platform and correlate there.

On Kubernetes, the container plugin enriches syscall events with k8s.pod.name, k8s.ns.name, container.name, and image metadata by talking to the node container runtime socket.

Select the deployment method

Upstream supports two common Kubernetes install paths:

This lab installs the Falco DaemonSet on every eligible cluster node. Add a nodeSelector only when you intentionally want to restrict the lab to named nodes. Production clusters should run the Falco DaemonSet on every node that runs workloads so syscall coverage is complete.

DaemonSet placement and privileges

Falco runs as a DaemonSet so each node gets one agent. The chart mounts host paths such as /proc, container runtime sockets, and (for eBPF) /sys/kernel. The Falco container needs elevated privileges to load the probe and read host namespaces.

Driver choice

Set driver.kind: modern_ebpf on kernels that support the modern eBPF probe. driver.kind: modern_ebpf forces the modern eBPF driver. Installation or startup fails if the node cannot support it. Use driver.kind: auto when you want Falco to try modern eBPF first and fall back to another supported driver. Verify node kernel compatibility before rollout — Rocky Linux 10.2 with kernel 6.12 in this lab supports modern eBPF.

Install with Helm

Add the chart repository on a workstation with kubectl and helm access:

helm repo add falcosecurity https://falcosecurity.github.io/charts

"falcosecurity" has been added to your local repository

Update indexes before install:

helm repo update

Create the falco namespace and install the chart. Save this initial falco-values.yaml:

tty: true

driver:
  kind: modern_ebpf

falco:
  json_output: true
  rule_matching: all

falcoctl:
  artifact:
    install:
      enabled: true
    follow:
      enabled: false
  config:
    artifact:
      install:
        refs:
          - falco-rules:5

collectors:
  enabled: true
  containerEngine:
    enabled: true

customRules files are mounted under /etc/falco/rules.d, and rule_matching: all allows both the stock rule and a later matching custom rule to emit alerts. The default first-match setting stops evaluation after the first matching rule.

helm upgrade --install falco falcosecurity/falco --version 9.1.0 --namespace falco --create-namespace --values falco-values.yaml --wait

When falcoctl artifact download works, the init container installs falco-rules:5 and the container plugin automatically. If the falcoctl-artifact-install init container cannot reach the artifact registry because of TLS inspection, mount the organization's CA certificate into that container. For a self-contained lab, disable artifact installation and following so Falco uses the rules bundled in the image:

falcoctl:
  artifact:
    install:
      enabled: false
    follow:
      enabled: false

Do not pre-stage files on the host unless the Helm values also define the corresponding hostPath and container mount.

Verify event collection

Do not run detection tests until Falco loads rules and opens the syscall source.

Check DaemonSet pods and node coverage:

kubectl get pods -n falco -o wide

NAME          READY   STATUS    RESTARTS   AGE   IP             NODE       NOMINATED NODE   READINESS GATES
falco-zctqt   1/1     Running   0          28s   192.168.1.17   worker01   <none>           <none>

A READY count of 1/1 on each scheduled node means the health endpoint responds. In production, confirm one Falco pod per worker (and control-plane nodes if they run workloads).

Read startup logs for driver, plugin, and rules:

kubectl logs --namespace falco --selector app.kubernetes.io/name=falco --container falco --tail=50

/etc/falco/falco_rules.yaml | schema validation: ok
Loaded event sources: syscall
Enabled event sources: syscall
Opening 'syscall' source with modern BPF probe.

The lines above confirm rules loaded, the syscall source is enabled, and the modern BPF probe opened successfully.

Test the health endpoint from the workstation:

POD=$(kubectl get pod -n falco -l app.kubernetes.io/name=falco -o jsonpath='{.items[0].metadata.name}')

kubectl port-forward -n falco "pod/$POD" 8765:8765 >/tmp/falco-port-forward.log 2>&1 &

PF_PID=$!
sleep 2

curl -fsS http://127.0.0.1:8765/healthz >/dev/null && echo "Falco health endpoint returned HTTP 200"

kill "$PF_PID"

Falco health endpoint returned HTTP 200

The Falco web server exposes health endpoints. Validate the HTTP result rather than depending on an undocumented exact response body.

Confirm the Falco version matches your chart:

kubectl exec -n falco "$POD" -c falco -- falco --version 2>&1 | head -1

2026-07-28T10:08:32+0000: Falco version: 0.44.1 (x86_64)

Inspect rules

Falco loads rules from ordered paths. On this Helm install:

Upstream publishes rules with maturity tags such as maturity_stable, maturity_incubating, and maturity_sandbox. Start with stable rules in production; promote incubating rules only after tuning false positives in your environment.

Do not edit packaged default rules in place. Add exceptions, macros, and new rules in /etc/falco/rules.d or through Helm customRules so upgrades do not overwrite your changes.

List loaded rule names from the rules file inside a Falco pod:

kubectl exec -n falco "$POD" -c falco -- grep -E '^- rule:' /etc/falco/falco_rules.yaml | head -10

- rule: Directory traversal monitored file read
- rule: Read sensitive file trusted after startup
- rule: Read sensitive file untrusted
- rule: Run shell untrusted
- rule: System user interactive
- rule: Terminal shell in container
- rule: Contact K8S API Server From Container
- rule: Netcat Remote Code Execution in Container
- rule: Search Private Keys or Passwords
- rule: Clear Log Activities

The exact rule names depend on the resolved falco-rules:5 artifact, or on the rules packaged with the Falco image when artifact installation is disabled. Record the loaded rules file and list rule names from the running Pod instead of assuming that either source always contains a larger rule set.

Detect shell execution

Administrative kubectl exec sessions spawn a shell inside a container. Falco rule Terminal shell in container fires when a shell starts with a TTY attached.

Create a lab namespace and sleep pod:

kubectl create namespace falco-lab

namespace/falco-lab created

kubectl run falco-test -n falco-lab --image=quay.io/curl/curl:8.5.0 --restart=Never --command -- sleep 3600

pod/falco-test created

Wait for the Pod to become Ready before running kubectl exec:

kubectl wait -n falco-lab --for=condition=Ready pod/falco-test --timeout=120s

Set the chart's top-level tty: true so Falco flushes alerts to container logs immediately during this lab. This setting does not create a terminal for monitored workloads. The stock rule still requires the target shell itself to have a TTY, which kubectl exec -it provides.

script -q -c 'kubectl exec -n falco-lab falco-test -it -- sh -c "echo shell-detection-test"' /dev/null

shell-detection-test

Tail Falco logs and locate the alert:

kubectl logs -n falco -l app.kubernetes.io/name=falco -c falco --since=2m | grep 'Terminal shell'

{"priority":"Notice","rule":"Terminal shell in container","output":"... process=sh ... terminal=34816 ... k8s_pod_name=falco-test k8s_ns_name=falco-lab ..."}

Read the fields together:

Legitimate break-glass admin sessions produce the same alert. Add namespace or image exceptions during tuning rather than disabling the rule cluster-wide.

Detect sensitive-file access

Use a disposable container path for this test. Reading /etc/shadow inside a throwaway Pod demonstrates the rule without touching host credentials on the node.

Deploy a root shell pod:

kubectl run shadow-test -n falco-lab --image=alpine:3.20 --restart=Never \
  --overrides='{"spec":{"containers":[{"name":"shadow-test","image":"alpine:3.20","command":["sleep","3600"],"securityContext":{"runAsUser":0}}]}}'

pod/shadow-test created

kubectl wait -n falco-lab --for=condition=Ready pod/shadow-test --timeout=120s

Attempt to read the shadow file:

kubectl exec -n falco-lab shadow-test -- sh -c 'cat /etc/shadow | head -1'

root:*::0:::::

Falco rule Read sensitive file untrusted should appear in logs:

kubectl logs -n falco -l app.kubernetes.io/name=falco -c falco --since=2m | grep 'Read sensitive file'

{"priority":"Warning","rule":"Read sensitive file untrusted","output":"... file=/etc/shadow ... process=cat ... k8s_pod_name=shadow-test k8s_ns_name=falco-lab ..."}

For environments that prefer not to touch /etc/shadow even in test Pods, create /tmp/sensitive-lab-test in a disposable container and extend a custom rule that watches that path.

Detect package installation

Package managers inside running containers often signal configuration drift on immutable workloads. Installing software writes new binaries that were not part of the image layer.

This test expects containerd to use the overlayfs snapshotter. With another snapshotter, proc.is_exe_upper_layer may not identify the installed executable and the stable rule may not fire. Falco also publishes separate package-manager rules at other maturity levels, but those may not be present in the stable falco-rules:5 set.

Deploy an Alpine pod and install a package:

kubectl run apk-test -n falco-lab --image=alpine:3.20 --restart=Never --command -- sleep 3600

pod/apk-test created

kubectl wait -n falco-lab --for=condition=Ready pod/apk-test --timeout=120s

kubectl exec -n falco-lab apk-test -- sh -c 'apk add --no-cache curl 2>&1 | tail -3'

Executing busybox-1.36.1-r31.trigger
Executing ca-certificates-20260413-r0.trigger
OK: 13 MiB in 24 packages

Falco rule Drop and execute new binary in container fires when a process executes a binary that was not present in the original image:

kubectl logs -n falco -l app.kubernetes.io/name=falco -c falco --since=5m | grep 'Drop and execute'

{"priority":"Critical","rule":"Drop and execute new binary in container","output":"... process=c_rehash ... k8s_pod_name=apk-test k8s_ns_name=falco-lab ..."}

The alert names the new executable (c_rehash during apk post-install hooks), the Pod, and the namespace. In production, pair this signal with image digest pinning and admission policy so drift is blocked before runtime.

Detect privilege-escalation activity

Run a controlled privileged Pod to trigger Debugfs Launched in Privileged Container. This demonstrates detection of dangerous tooling in privileged contexts — not host escape.

kubectl run priv-rocky -n falco-lab --image=quay.io/rockylinux/rockylinux:9 --restart=Never \
  --overrides='{"spec":{"containers":[{"name":"priv-rocky","image":"quay.io/rockylinux/rockylinux:9","command":["sleep","3600"],"securityContext":{"privileged":true,"runAsUser":0}}]}}'

pod/priv-rocky created

kubectl wait -n falco-lab --for=condition=Ready pod/priv-rocky --timeout=120s

Install e2fsprogs and run debugfs. The Falco rule only needs to observe debugfs launched in a privileged container — it does not require a specific block device path:

kubectl exec -n falco-lab priv-rocky -- sh -c 'dnf install -y e2fsprogs >/dev/null && /sbin/debugfs -V 2>&1 | head -1'

debugfs 1.46.5 (30-Dec-2021)

The version text varies, so your output may show debugfs <version>.

Falco emits a Warning for the privileged debugfs execution:

kubectl logs -n falco -l app.kubernetes.io/name=falco -c falco --since=5m | grep Debugfs

{"priority":"Warning","rule":"Debugfs Launched in Privileged Container","output":"... process=debugfs ... command=debugfs -V ... k8s_pod_name=priv-rocky k8s_ns_name=falco-lab ..."}

Delete privileged test Pods immediately after validation. Prefer Pod Security Standards and restricted securityContext in production so privileged workloads cannot start without an explicit exception.

Write a focused custom rule

Add a narrow rule for TTY-attached shell activity in a protected namespace. Save falco-custom-values.yaml:

customRules:
  custom-falco.yaml: |-
    - rule: Shell in protected namespace
      desc: Detect a TTY-attached shell in falco-lab-protected
      condition: >
        spawned_process and container and
        k8s.ns.name = "falco-lab-protected" and
        proc.tty != 0 and
        proc.name in (bash, sh, zsh, ash, dash)
      output: >
        Protected namespace shell
        (user=%user.name ns=%k8s.ns.name pod=%k8s.pod.name
        container=%container.name proc=%proc.name cmdline=%proc.cmdline)
      priority: WARNING
      tags: [custom, falco-lab]

    - rule: Terminal shell in container
      exceptions:
        - name: admin_debug_namespaces
          fields: k8s.ns.name
          comps: in
          values:
            - falco-lab
      override:
        exceptions: append

Falco requires the overriding file to load after the default rules file, and override.exceptions: append adds the exception without replacing existing rule exceptions.

Apply the custom rule with a second Helm values file:

helm upgrade falco falcosecurity/falco \
  --version 9.1.0 \
  --namespace falco \
  --values falco-values.yaml \
  --values falco-custom-values.yaml \
  --wait

kubectl rollout status daemonset/falco --namespace falco --timeout=180s

After the upgrade, startup logs should include the custom rules file:

/etc/falco/rules.d/custom-falco.yaml | schema validation: ok

Create the protected namespace and test Pod:

kubectl create namespace falco-lab-protected

namespace/falco-lab-protected created

kubectl run protected-shell -n falco-lab-protected --image=alpine:3.20 --restart=Never --command -- sleep 3600

pod/protected-shell created

kubectl wait -n falco-lab-protected --for=condition=Ready pod/protected-shell --timeout=120s

script -q -c 'kubectl exec -n falco-lab-protected protected-shell -it -- sh -c "echo custom-rule-test"' /dev/null

Verify the custom rule fired:

Pattern matching uses grep command.

kubectl logs -n falco -l app.kubernetes.io/name=falco -c falco --since=2m | grep '"rule":"Shell in protected namespace"'

{"priority":"Warning","rule":"Shell in protected namespace","output":"... k8s_ns_name=falco-lab-protected k8s_pod_name=protected-shell ..."}

With rule_matching: all in falco-values.yaml, both the stock Terminal shell in container rule and this custom rule can emit alerts on the same event. Grep for the custom rule name to confirm it loaded and matched.

Validate and tune

Work through four tuning passes on a rule you care about.

Positive case

Confirm the custom rule or stock rule fires on the test action you designed. If nothing appears, verify event collection, rule syntax, and priority threshold (falco.priority in config filters rules at load time by minimum priority).

Expected normal activity

Run the same action in an admin namespace (falco-lab) and confirm whether the alert is acceptable. Break-glass kubectl exec in platform namespaces is often expected noise.

Namespace exception

The falco-custom-values.yaml block above already appends a namespace exception to Terminal shell in container for the admin namespace falco-lab. Apply it with Helm if you have not already:

helm upgrade falco falcosecurity/falco \
  --version 9.1.0 \
  --namespace falco \
  --values falco-values.yaml \
  --values falco-custom-values.yaml \
  --wait

kubectl rollout status daemonset/falco --namespace falco --timeout=180s

Verify that the stock rule is suppressed in falco-lab while the protected-namespace custom rule still fires in falco-lab-protected:

script -q -c 'kubectl exec -n falco-lab falco-test -it -- sh -c "echo exception-test"' /dev/null

sleep 2

if kubectl logs -n falco -l app.kubernetes.io/name=falco -c falco --since=10s | grep -q '"rule":"Terminal shell in container".*k8s_ns_name=falco-lab'; then
  echo "Unexpected alert: exception did not apply"
else
  echo "Exception suppressed the stock rule in falco-lab"
fi

Exception suppressed the stock rule in falco-lab

Without the override directive, a later entry may replace rather than append to the packaged rule definition.

Validate YAML with falco --validate or watch Falco logs after reload. A malformed exception prevents Falco from starting.

Image and parent-process exceptions

Narrow false positives further with image repository tags or parent process names when a specific controller spawns shells during normal operation. Prefer scoped exceptions over enabled: false on broad rules.

Route alerts

Falco can emit alerts to several sinks configured in falco.yaml:

This lab enables JSON on stdout in falco-values.yaml for easy parsing. Add more sinks when you are ready to route alerts beyond kubectl logs:

falco:
  stdout_output:
    enabled: true
  syslog_output:
    enabled: true

A full alert pipeline — deduplication, routing on severity, on-call paging, and retention — belongs in Falcosidekick or your observability stack. Wire outputs after detection works; do not block the first rollout on SIEM integration.

Understand detection limits

Falco is a detection tool, not a complete security program. It does not:

• Prevent all malicious actions by itself
• Replace Kubernetes audit logs or application logs
• Scan container images for CVEs
• Reconstruct incidents automatically across all data sources
• Prove that an alert is malicious without analyst context
• Correlate syscall, audit, and cloud events into one incident without external tooling

Use Falco together with audit logging, image scanning, admission policy, and network controls from the security architecture layers.

Troubleshooting

When container-plugin errors appear, inspect the init container and Falco logs:

POD=$(kubectl get pod -n falco -l app.kubernetes.io/name=falco -o jsonpath='{.items[0].metadata.name}')

kubectl logs -n falco "$POD" -c falcoctl-artifact-install

kubectl logs -n falco "$POD" -c falco | grep -iE 'container|plugin'

What's Next

• Kubernetes Incident Response and Forensics

References

• Falco documentation
• Falco rules repository
• Falco Helm chart
• Falco Operator
• Falcosidekick
• Container plugin configuration

Summary

You deployed Falco on Kubernetes with the Helm chart and modern eBPF driver, confirmed that syscall events and rules loaded before testing, and walked through four controlled detections: terminal shell in a container, sensitive-file read, package installation drift, and debugfs inside a privileged Pod. Each alert tied a rule name and priority to Kubernetes metadata such as namespace, Pod, container, process, and command line.

The important operational lesson is to verify collection first. A crashing driver, missing container plugin, or invalid custom rule file produces silence or CrashLoopBackOff, not trustworthy detection. Add local rules and exceptions in /etc/falco/rules.d rather than editing packaged defaults, and scope exceptions to admin namespaces or known images when break-glass kubectl exec is normal in your environment.

Falco tells you that something matched a rule at runtime; it does not replace audit logs, image scanning, or admission controls. Route JSON alerts through stdout, syslog, or Falcosidekick when you are ready, and pair runtime signals with immutable workload design and API audit evidence for defense in depth on production clusters.