Harden Linux Nodes for Kubernetes

A compromised Linux node can expose every workload on that host, mounted volumes, ServiceAccount tokens, and the container runtime socket. This checklist inventories a kubeadm lab control-plane node and worker running Cilium, walks through tested hardening steps where commands are shown, and gives you inspection points for categories you apply in your own change windows.

Use a separate administration workstation or the control-plane node for kubectl commands. Do not use the worker being drained as the administration host. This lab runs kubectl from k8s-cp; OS commands run over SSH on the node being hardened. Expected durations are noted for steps that take more than a minute.

This lab successfully ran Cilium 1.19.6 with Kubernetes 1.36.3. Kubernetes 1.36 is not listed in the Cilium 1.19.6 guaranteed compatibility matrix, so verify the supported version combination before reproducing this outside the lab.

Define the node security boundary

The node is more than the kubelet process. Host compromise may expose:

• Running workloads and their process namespaces
• Mounted volumes and hostPath data
• ServiceAccount tokens mounted into Pods
• The container runtime socket (containerd.sock or equivalent)
• Kubelet client certificates and kubeconfig files
• CNI and CSI plugin credentials on disk
• Cloud instance metadata and node IAM roles
• The host kernel and any privileged containers

Kubernetes policy limits what a stolen API token can do cluster-wide. A root-equivalent foothold on the node can bypass that boundary by reading files, creating containers directly through the runtime socket, or reading node-local credentials. Node hardening shrinks that blast radius before you tune workload controls.

Inventory the node

Record a baseline on every control-plane and worker node before you change anything. SSH to the node and capture OS identity:

hostname
grep PRETTY_NAME /etc/os-release
uname -r
kubelet --version
containerd --version | head -1

k8s-cp
PRETTY_NAME="Rocky Linux 10.2 (Red Quartz)"
6.12.0-211.16.1.el10_2.0.1.x86_64
Kubernetes v1.36.3
containerd containerd.io 2.2.5 1.el10_2

List listening TCP sockets to see what faces the network:

ss -lntp

On the lab control plane, 6443 (API server), 10250 (kubelet), 10256 (kube-proxy health), and 22 (SSH) listen on all interfaces. etcd listens on 192.168.56.108:2379 and 2380. Scheduler and controller-manager ports bind to 127.0.0.1 only—a healthier pattern than wide exposure.

Count installed packages and note compilers you may remove later:

rpm -qa | wc -l
rpm -q gcc make 2>&1

693
package gcc is not installed
package make is not installed

Store inventory in a change ticket: OS version, kernel, enabled repositories, sudoers, SSH effective settings, firewall zone, CNI mode, and Kubernetes paths under /etc/kubernetes and /var/lib/kubelet.

Reduce installed packages

Remove software that expands the attack surface without supporting cluster operations:

• Compilers and build toolchains on production nodes
• Legacy remote-access agents you no longer use
• Desktop environments and debug servers
• Duplicate package repositories pointing at untrusted mirrors

Retain tools your runbooks require—tcpdump for incident response, chrony for clock sync, and the container runtime packages themselves.

On this lab control plane, gcc and make are already absent. Before you run dnf remove on a worker, confirm no host agent or monitoring installer still needs the package. Uninstall in a maintenance window and reboot only when the package list changes kernel modules.

Disable unused services

SSH to the node you are hardening. List running and enabled units:

systemctl --type=service --state=running --no-pager | head -15

systemctl list-unit-files --state=enabled --type=service --no-pager | head -15

On the control plane, expected long-running services include kubelet, containerd, firewalld, chronyd, and sshd. The lab worker also had httpd enabled from other tutorials—a candidate to disable when that node is Kubernetes-only.

Before you systemctl disable --now a service, document why it listens on the network. Ask whether Kubernetes, storage, monitoring, or LDAP still needs it. Disabling firewalld or chronyd on a node causes different failures than removing httpd; validate cluster health after each disable.

On worker01, disable Apache when the node no longer serves HTTP outside the cluster:

systemctl disable --now httpd

Confirm the unit stays off across reboot:

systemctl is-enabled httpd

disabled

systemctl is-active httpd

inactive

The worker no longer starts httpd at boot. Re-enable only if a workload or runbook still needs it on that host.

Secure administrative access

Inspect effective SSH settings—the file comments may hide defaults:

sshd -T | grep -E 'permitrootlogin|passwordauthentication|pubkeyauthentication'

permitrootlogin yes
pubkeyauthentication yes
passwordauthentication yes

This lab still allows root login and password authentication. Production hardening typically moves to:

• Dedicated admin users in the wheel or sudo group
• Key-based SSH only (PasswordAuthentication no)
• PermitRootLogin prohibit-password or no
• Source restrictions in firewalld rich rules or security-group equivalents
• Session recording for break-glass accounts

Apply SSH changes through a drop-in under /etc/ssh/sshd_config.d/ so distribution updates do not overwrite your edits. This example caps authentication attempts without touching root login yet:

cat > /etc/ssh/sshd_config.d/99-glc-hardening.conf <<'EOF'
# GoLinuxCloud node hardening example
MaxAuthTries 4
EOF

Validate syntax before reload—a bad file can lock you out on the next restart:

sshd -t && echo "sshd configuration valid"

sshd configuration valid

Reload SSH and confirm the effective value:

systemctl reload sshd

sshd -T | grep maxauthtries

maxauthtries 4

Change SSH in one step, keep a console session open, and test from a second terminal before you close your path in. Shared root passwords and permanent PermitRootLogin yes fail most security reviews.

Limit sudo access

Separate who may patch the OS, change firewalld, restart kubelet, or run kubectl with cluster-admin credentials.

Example sudo split (illustrative—adapt to your org):

List custom sudoers files before you add new ones:

ls -la /etc/sudoers.d/

An empty directory means only the main /etc/sudoers rules apply. Prefer visudo fragments per team instead of handing out unrestricted root.

Inspect SELinux and swap

On RHEL-family nodes, confirm SELinux mode and swap before you treat the host as production-ready.

getenforce

Permissive

This node does not yet meet the intended SELinux hardening target. Investigate denials and test Kubernetes and Cilium under Enforcing before marking this category complete.

SELinux should normally stay Enforcing on hardened nodes. Permissive or Disabled modes remove a major containment layer; fix policy violations rather than leaving the host permissive indefinitely.

Check whether swap is active:

swapon --show

An empty result means no swap devices are enabled on this worker. When swap is present, memory-backed Secret data can reach persistent swap storage under unsupported configurations. Do not assume swap is safe or automatically blocked—verify kubelet swap settings and kernel support for your Kubernetes version before you leave swap enabled.

Configure host firewall paths

The Cilium validation steps later in this section use the Cilium CLI (cilium), which is separate from the in-cluster Cilium DaemonSet. Install it on your administration host before you run cilium status or cilium connectivity test—see Cilium CLI installation.

On k8s-cp, confirm the client version matches the lab:

cilium version --client

cilium-cli: v0.19.6 compiled with go1.26.4 on linux/amd64
cilium image (default): v1.19.5
cilium image (stable): v1.19.6

Map required ports per node role before you remove rules. The lab uses firewalld on Rocky Linux with Cilium in VXLAN tunnel mode.

The host-firewall procedure must match the installed CNI. Cilium defaults to 8472/udp for VXLAN and 6081/udp for Geneve, but the tunnel port is configurable. Confirm the effective configuration from the cilium-config ConfigMap before you open host ports. A firewall-cmd --reload can disrupt CNI-managed rules; validate the Cilium data plane after every reload on nodes where firewalld stays enabled.

From k8s-cp (or your separate admin workstation), read the tunnel settings:

kubectl -n kube-system get cm cilium-config -o jsonpath='tunnel-protocol={.data.tunnel-protocol}{"\n"}tunnel-port={.data.tunnel-port}{"\n"}'

tunnel-protocol=vxlan
tunnel-port=

An empty tunnel-port means Cilium uses the default VXLAN port 8472/udp. If your cluster sets a custom tunnel-port, open that UDP port instead.

Control-plane ingress (this cluster):

Do not list 10257/tcp or 10259/tcp as required network ingress when the controller-manager and scheduler bind to 127.0.0.1 only, as on this lab.

Cilium tunnel ports (open only the mode you use):

Optional Cilium health connectivity:

Cilium continues operating without these health paths, but connectivity-health reporting becomes incomplete.

Worker ingress (subset):

SSH to each Cilium node and review the active zone before you change ports:

firewall-cmd --list-all

public (active)
  services: cockpit dhcpv6-client ntp ssh
  ports: 1313/tcp 6443/tcp 2379-2380/tcp 10250/tcp 10259/tcp 10257/tcp 10256/tcp 30000-32767/tcp 4789/udp 5473/tcp 179/tcp

The zone still lists Calico-era ports (4789, 179, 5473) and localhost-only controller ports from an earlier setup pass. This cluster runs Cilium VXLAN, so add 8472/udp and remove stale entries on every node.

Apply the CNI correction on each node (about one minute per node including reload). Remove 1313/tcp if nothing on the node listens there—this lab had that port left over from an earlier Hugo preview with no current listener:

firewall-cmd --permanent --add-port=8472/udp
firewall-cmd --permanent --remove-port=4789/udp
firewall-cmd --permanent --remove-port=179/tcp
firewall-cmd --permanent --remove-port=5473/tcp
firewall-cmd --permanent --remove-port=10257/tcp
firewall-cmd --permanent --remove-port=10259/tcp
firewall-cmd --permanent --remove-port=1313/tcp
firewall-cmd --permanent --remove-service=cockpit
firewall-cmd --reload
firewall-cmd --list-services
firewall-cmd --list-ports

Control-plane sample after reload:

dhcpv6-client ntp ssh
2379-2380/tcp 6443/tcp 10250/tcp 10256/tcp 30000-32767/tcp 8472/udp

A worker should not normally list control-plane-only entries such as 6443/tcp or 2379-2380/tcp. On worker01 after the same CNI correction (this lab also still has LDAP tutorial ports):

dhcpv6-client ldap ssh
1636/tcp 3490/tcp 3590/tcp 10250/tcp 10256/tcp 30000-32767/tcp 8472/udp

Removing cockpit from the services list is what proves that service was dropped—--list-ports alone cannot show a removed service.

Node readiness alone does not prove cross-node Pod networking survived a firewall change. From k8s-cp, confirm Cilium is healthy (about 30 seconds):

cilium status --wait

Relevant output:

/¯¯\
 /¯¯\__/¯¯\    Cilium:             OK
 \__/¯¯\__/    Operator:           OK
 /¯¯\__/¯¯\    Envoy DaemonSet:    OK
 \__/¯¯\__/    Hubble Relay:       disabled
    \__/       ClusterMesh:        disabled

DaemonSet              cilium                   Desired: 2, Ready: 2/2, Available: 2/2
...
Cluster Pods:          7/7 managed by Cilium
Helm chart version:    1.19.6

Run a basic Pod networking check

Use an isolated namespace so Istio sidecars do not interfere, and clean up synchronously so reruns do not hit name collisions:

kubectl create namespace node-hardening-test
kubectl label namespace node-hardening-test \
  istio-injection=disabled --overwrite
kubectl -n node-hardening-test run net-a --image=nginx:alpine \
  --restart=Never --port=80
kubectl -n node-hardening-test wait --for=condition=Ready pod/net-a \
  --timeout=90s
IP=$(kubectl -n node-hardening-test get pod net-a \
  -o jsonpath='{.status.podIP}')
kubectl -n node-hardening-test run net-b --rm -i --restart=Never \
  --image=busybox:1.36 --timeout=60s -- wget -qO- --timeout=5 \
  http://${IP} | head -2
kubectl delete namespace node-hardening-test --wait=true

<!DOCTYPE html>
<html>

This confirms that Pod-IP networking still works on the schedulable worker. It does not validate inter-node VXLAN because both test Pods may run on the same node in a single-worker lab. Cilium's inter-host connectivity suite is what exercises the overlay tunnel between nodes—--single-node intentionally skips those cases.

For deeper Cilium validation, cilium connectivity test deploys a temporary test namespace and runs many scenarios. Expect 8–15 minutes for a full run on a two-node cluster, or 5–8 minutes with --single-node on a single-worker lab. Always pass an explicit suite timeout:

cilium connectivity test --single-node --timeout 8m

If external HTTPS or DNS egress is restricted in your lab, the suite may hang on pod-to-world scenarios—fix cluster DNS upstream forwarders or scope tests with --test filters. A full multi-node run also needs enough schedulable disk on every node; a control plane under disk pressure will evict test Pods.

Restrict outbound access

Inventory destinations nodes must reach before you block egress:

• API server https://<cp>:6443 from workers
• DNS resolvers and NTP
• OS package mirrors or internal patch servers
• Container image registries (registry.k8s.io, mirrors)
• Storage backends (NFS, iSCSI, cloud APIs)
• Monitoring and log collectors

Blanket deny-all egress breaks image pulls and webhook callbacks. Document allowed FQDNs or CIDRs per node role, implement in host firewall or network policy at the hypervisor layer, and watch for ImagePullBackOff or NodeNotReady after each change.

Protect Kubernetes files

Sensitive paths on kubeadm nodes include:

/etc/kubernetes/          — admin kubeconfig, PKI, manifests
/var/lib/kubelet/       — kubelet config, pod state
/var/lib/etcd/          — etcd data (control plane)
/etc/cni/net.d/         — CNI configuration
/run/containerd/        — runtime socket

SSH to the node and audit ownership and modes.

Run on every node:

ls -ld /etc/kubernetes /var/lib/kubelet /etc/cni/net.d
ls -l /var/lib/kubelet/config.yaml

Run on control-plane nodes only:

ls -ld /var/lib/etcd
ls -l /etc/kubernetes/admin.conf /etc/kubernetes/pki/ca.key

Control-plane sample:

drwxr-xr-x. 5 root root 4096 Jul 23 00:02 /etc/kubernetes
drwxr-xr-x. 10 root root 4096 Jul 28 03:20 /var/lib/kubelet
drwx------. 3 root root 4096 Jul 27 22:21 /var/lib/etcd
-rw-------. 1 root root 5638 /etc/kubernetes/admin.conf
-rw-------. 1 root root 1679 /etc/kubernetes/pki/ca.key
-rw-r--r--. 1 root root 1118 /var/lib/kubelet/config.yaml

admin.conf and ca.key are already 600. The kubelet configuration was world-readable (644)—kube-bench flags that on CIS check 4.1.9.

Tighten the kubelet config mode on the control plane:

chmod 600 /var/lib/kubelet/config.yaml
stat -c '%a %n' /var/lib/kubelet/config.yaml

600 /var/lib/kubelet/config.yaml

Confirm kubelet keeps running:

systemctl is-active kubelet

active

Repeat on workers. The lab worker already had 600 on /var/lib/kubelet/config.yaml from an earlier CIS remediation pass.

Protect runtime sockets

The container runtime socket grants host-level container control. On this lab:

ls -l /run/containerd/containerd.sock

srw-rw----. 1 root root 0 Jul 27 22:10 /run/containerd/containerd.sock

Mode 660 with group root limits access to root-equivalent users. Anyone who can write to the socket can pull images and create privileged containers.

Reduce risk by:

• Keeping mode 660 or tighter and avoiding broad group membership
• Denying remote access to CRI endpoints unless encrypted and authenticated
• Preventing workload users from joining the docker or containerd groups

Socket hardening pairs with node metadata and kubelet endpoint controls at the Kubernetes layer.

Patch nodes safely

OS patches on Kubernetes nodes follow a rolling pattern. From k8s-cp:

kubectl cordon worker01

node/worker01 cordoned

Cordon marks the node unschedulable while existing Pods keep running:

kubectl get node worker01

NAME       STATUS                     ROLES    AGE    VERSION
worker01   Ready,SchedulingDisabled   <none>   5h8m   v1.36.3

List workloads still bound to the node before you drain:

kubectl get pods -A -o wide --field-selector spec.nodeName=worker01

NAMESPACE      NAME                              READY   STATUS    RESTARTS   AGE     IP               NODE
istio-system   istio-cni-node-cntjf              1/1     Running   0          5h48m   192.168.1.67     worker01
istio-system   istiod-68b8c6d945-57sd2           1/1     Running   0          5h50m   192.168.1.140    worker01
kube-system    cilium-envoy-cvm65                1/1     Running   0          7h9m    192.168.56.109   worker01
kube-system    cilium-fdkjr                      1/1     Running   0          5h43m   192.168.56.109   worker01
kube-system    coredns-589f44dc88-bg4dm          1/1     Running   0          5h45m   192.168.1.21     worker01
kube-system    coredns-589f44dc88-dc4j9          1/1     Running   0          5h45m   192.168.1.203    worker01
kube-system    kube-proxy-xnmv9                  1/1     Running   0          14h     192.168.56.109   worker01

Drain evicts workloads before you reboot. The --delete-emptydir-data flag permanently removes Pod emptyDir data—confirm no application still needs that ephemeral storage. PodDisruptionBudgets can block eviction until enough replicas run elsewhere. Do not start OS patching or reboot until kubectl drain exits successfully. Expect 1–5 minutes for drain on a lightly loaded worker; PDB conflicts can extend that.

kubectl drain waits indefinitely by default. Use an explicit --timeout so a PDB-blocked drain exits instead of retrying forever.

First, run a short diagnostic drain to surface the PDB conflict:

kubectl drain worker01 \
  --ignore-daemonsets \
  --delete-emptydir-data \
  --timeout=60s

node/worker01 already cordoned
Warning: ignoring DaemonSet-managed Pods: istio-system/istio-cni-node-cntjf, kube-system/cilium-envoy-cvm65, kube-system/cilium-fdkjr, kube-system/kube-proxy-xnmv9
evicting pod istio-system/istiod-68b8c6d945-6k9nl
error when evicting pods/"istiod-68b8c6d945-6k9nl" -n "istio-system" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
...
There are pending pods in node "worker01" when an error occurred: error when evicting pods/"istiod-68b8c6d945-6k9nl" -n "istio-system": global timeout reached: 1m0s
error: unable to drain node "worker01" due to error: ... global timeout reached: 1m0s, continuing command...

The command exits non-zero after the timeout when the PDB remains unsatisfied. That drain did not finish—do not patch or reboot while drain still reports errors.

This single-worker lab cannot demonstrate a PDB-respecting, no-downtime drain unless another schedulable node is available. Add another worker or explicitly accept workload downtime.

To complete drain on this lab, scaling istiod to zero directly removes its Pod and explicitly accepts an Istio control-plane outage; it does not make the eviction satisfy the PDB. Preserve the original replica count before you scale:

ISTIOD_REPLICAS=$(
  kubectl get deployment istiod -n istio-system \
    -o jsonpath='{.spec.replicas}'
)

kubectl scale deployment istiod -n istio-system --replicas=0

Re-run drain with a bounded timeout and confirm it exits successfully:

kubectl drain worker01 \
  --ignore-daemonsets \
  --delete-emptydir-data \
  --timeout=5m

node/worker01 already cordoned
Warning: ignoring DaemonSet-managed Pods: istio-system/istio-cni-node-cntjf, kube-system/cilium-envoy-cvm65, kube-system/cilium-fdkjr, kube-system/kube-proxy-xnmv9
node/worker01 drained

Only after node/worker01 drained appears may you patch with dnf upgrade on the node over SSH and reboot if the kernel changed. After reboot, wait for the node to report Ready (up to five minutes).

On worker01 over SSH:

systemctl is-active kubelet containerd

active
active

On k8s-cp:

kubectl wait --for=condition=Ready node/worker01 --timeout=5m

node/worker01 condition met

Uncordon when the node is healthy:

kubectl uncordon worker01

node/worker01 uncordoned

Confirm the node is Ready and schedulable. Uncordoning permits future scheduling but does not automatically rebalance existing Pods back to this node.

kubectl get node worker01

NAME       STATUS   ROLES    AGE   VERSION
worker01   Ready    <none>   15h   v1.36.3

Restore scaled-down workloads when maintenance ends:

kubectl scale deployment istiod -n istio-system \
  --replicas="${ISTIOD_REPLICAS}"

kubectl rollout status deployment/istiod \
  -n istio-system --timeout=5m

deployment "istiod" successfully rolled out

kubectl get pods -n istio-system -o wide

NAME                      READY   STATUS    RESTARTS   AGE   IP              NODE
istio-cni-node-cntjf      1/1     Running   0          7h    192.168.1.67    worker01
istiod-68b8c6d945-6k9nl   1/1     Running   0          10m   192.168.1.133   worker01

Repeat one node at a time. For control-plane nodes, follow your HA runbook—etcd quorum and API availability matter more than on a single worker. See upgrade a Kubernetes cluster for version skew rules that also apply during long maintenance windows.

If you copied admin.conf to a worker during an earlier lab pass, remove it when you finish—admin.conf grants cluster-admin access and should stay on the control plane or a dedicated admin workstation:

rm -f /root/.kube/config   # on the worker, not on k8s-cp

Validate the hardened node

After each control category, rerun a short checklist.

On the hardened node over SSH:

ss -lntp | grep -E ':22|:6443|:10250'
systemctl is-active kubelet containerd firewalld

On k8s-cp (or your separate admin workstation):

kubectl get nodes

Control-plane sample:

LISTEN 0 4096 *:10250 *:* users:(("kubelet",...))
LISTEN 0 4096 *:6443 *:* users:(("kube-apiserver",...))
active
active
active
NAME       STATUS   ROLES           AGE     VERSION
k8s-cp     Ready    control-plane   5h10m   v1.36.3
worker01   Ready    <none>          5h8m    v1.36.3

Recheck file modes on paths you changed, SSH login from an admin jump host, and Pod networking with a test workload if you altered firewall or CNI ports. Run kube-bench periodically to catch permission drift.

Hardening quick reference

Use this table to correlate inspection commands with expected results during study or audits:

Troubleshooting

What's Next

• Use seccomp Profiles in Kubernetes
• Use AppArmor Profiles in Kubernetes
• Kubernetes SecurityContext with Practical Examples

References

• Kubernetes Ports and Protocols — control-plane and node port reference
• Communication between Nodes and the Control Plane — kubelet and API connectivity
• Safely drain a node (kubectl drain) — eviction, PDBs, and flags
• Security For Linux Nodes — upstream node hardening guidance
• Creating a cluster with kubeadm — admin.conf, cluster-admin scope, and external administration
• kubeadm implementation details — where admin.conf belongs and why it should not be shared
• Cilium system requirements — CNI ports and host firewall interaction
• Cilium end-to-end connectivity testing — connectivity-suite behavior, test selection, and cleanup
• firewalld documentation — zone and rich rules
• OpenSSH server configuration — sshd_config reference

Summary

You mapped the Linux node as a security boundary that holds workloads, tokens, runtime sockets, and platform credentials—not just the kubelet binary. The inventory pass recorded OS versions, listening ports, Cilium tunnel mode, package count, SSH effective settings, and permission modes on /etc/kubernetes and /var/lib/kubelet before any change.

Tested steps included disabling an unused httpd service, applying an SSH drop-in validated with sshd -t, aligning firewalld ports with Cilium VXLAN on every node, and running a basic Pod-IP check in an isolated namespace after reload. The patch workflow listed Pods on the node, showed a PDB-blocked drain with --timeout=60s on this single-worker cluster, scaled istiod to accept downtime while preserving the original replica count, completed drain with node/worker01 drained, then restored the Deployment with rollout status. Run kubectl from the control plane or a separate admin workstation—not from the worker being drained.

Match host firewall rules to your CNI tunnel mode—this lab uses Cilium VXLAN on 8472/udp, not Calico ports left from an earlier zone. Blocking kubelet port 10250 from the control plane breaks logs, exec, and port-forwarding; it does not normally flip node readiness, which kubelet reports through its outbound API connection. Pair this checklist with API server controls and periodic kube-bench scans so the OS and Kubernetes layers stay aligned.