Protect Kubernetes Node Metadata and Kubelet Endpoints

A compromised Pod that can reach the kubelet API or a cloud metadata service may steal credentials without ever passing your API RBAC checks. This walkthrough maps those two paths on a kubeadm lab, inspects kubelet settings in /var/lib/kubelet/config.yaml, tests authentication and nodes/proxy access, and records what ordinary identities can reach.

Map the node access paths

Attackers can reach node-level interfaces through the API or through the network:

Two paths matter in almost every review:

RBAC on the Kubernetes API does not automatically protect a kubelet port that is open on the node network. Treat authentication, authorization, firewall rules, and nodes/proxy bindings as one review.

Discover the kubelet configuration

kubeadm stores kubelet settings in /var/lib/kubelet/config.yaml. Systemd passes --config=/var/lib/kubelet/config.yaml through /etc/kubernetes/kubelet.conf bootstrap wiring — read the file instead of guessing from process arguments alone.

On a worker node, print the authentication and authorization block:

grep -E '^(authentication|authorization|kind):|anonymous|webhook|mode|clientCA|readOnly|port|address' /var/lib/kubelet/config.yaml

authentication:
  anonymous:
  webhook:
authorization:
  mode: Webhook
  webhook:
kind: KubeletConfiguration

Confirm which port kubelet listens on:

ss -tlnp | grep kubelet

LISTEN 0 4096 127.0.0.1:10248 0.0.0.0:* users:(("kubelet",pid=781683,fd=25))
LISTEN 0 4096 *:10250 0.0.0.0:* users:(("kubelet",pid=781683,fd=17))

This lab kubelet binds the secure API on 10250 on all interfaces (*:10250). The healthz endpoint for monitoring sits on 127.0.0.1:10248. Serving certificates are issued to the node name — here subject=CN=worker01@... from the kubelet server cert.

Other fields to note in the same file:

• authentication.x509.clientCAFile — which CA validates client certificates
• containerRuntimeEndpoint — CRI socket path (separate hardening topic)
• Legacy readOnlyPort — when absent or zero, the unauthenticated read-only port should be off (verify with a connection test below)

Disable anonymous authentication

kubeadm defaults should leave anonymous kubelet access disabled. Confirm in config:

authentication:
  anonymous:
    enabled: false

Test an unauthenticated request to the secure port on the node:

curl -sk https://127.0.0.1:10250/pods -o /dev/null -w "HTTP %{http_code}\n"

HTTP 401

The body prints Unauthorized. That is the expected baseline — the kubelet answered but rejected the caller.

Repeat from another host on the node network (control plane to worker in this lab):

curl -sk https://192.168.56.109:10250/healthz

Unauthorized

Even when anonymous auth is disabled, a reachable port still leaks existence of the API. Combine authentication settings with network restrictions in the next sections.

Enable webhook token authentication

Bearer tokens from ServiceAccounts or kubectl create token authenticate through the API server TokenReview API when webhook authentication is enabled:

authentication:
  webhook:
    enabled: true

Test a token that authenticates but should not be authorized. On the control plane:

export KUBECONFIG=/etc/kubernetes/admin.conf
TOKEN=$(kubectl create token default -n default --duration=1h)
curl -sk -H "Authorization: Bearer ${TOKEN}" https://192.168.56.109:10250/pods -o /dev/null -w "HTTP %{http_code}\n"

HTTP 403

401 means authentication failed. 403 means the kubelet accepted the identity and webhook authorization denied the verb — the desired split for a low-privilege token.

The API server and monitoring agents that legitimately call the kubelet use client certificates or identities bound to system:kubelet-api-admin (see below). After you disable weaker methods, confirm those agents still pass health checks.

Use webhook authorization

With kubeadm, authorization mode is typically:

authorization:
  mode: Webhook

The kubelet sends SubjectAccessReview requests to the API server. This is not an admission webhook — it runs on the node for each kubelet HTTP request after authentication succeeds.

A kubelet client certificate alone does not bypass this layer. Testing with the node's own client cert from localhost:

curl -sk --cert /var/lib/kubelet/pki/kubelet-client-current.pem --key /var/lib/kubelet/pki/kubelet-client-current.pem --cacert /etc/kubernetes/pki/ca.crt https://127.0.0.1:10250/healthz

Forbidden (user=system:node:worker01, verb=get, resource=nodes, subresource(s)=[healthz proxy])

The node identity is recognized, but the kubelet still enforces API authorization for the subresource.

Verify the read-only port status

Older clusters exposed an unauthenticated read-only kubelet on port 10255. Modern kubeadm builds omit it — verify with a connection attempt rather than assuming:

curl -sk http://127.0.0.1:10255/pods -o /dev/null -w "readOnly HTTP %{http_code}\n"

readOnly HTTP 000

HTTP 000 means nothing is listening. If you see 200 on 10255, treat it as a critical finding and disable the read-only port in kubelet configuration.

Restrict direct network reachability

Even with authentication enabled, limit who can open TCP connections to port 10250.

On this lab, firewalld on the worker allows 10250/tcp from broad sources. Production baselines usually narrow sources to:

• Control-plane nodes that call the kubelet through the API
• Approved monitoring collectors
• Administrative jump networks

List open ports:

firewall-cmd --list-ports

179/tcp 1636/tcp 2379-2380/tcp ... 10250/tcp 10256/tcp ...

Pair host firewall rules with cloud security groups or network ACLs when nodes run in a VPC. NetworkPolicy controls Pod-to-Pod traffic; it does not replace blocking Pod CIDRs from reaching nodeIP:10250 when the CNI allows node routing.

Audit nodes/proxy permissions

API clients with nodes/proxy can tunnel to the kubelet. Inspect the built-in system:kubelet-api-admin ClusterRole:

export KUBECONFIG=/etc/kubernetes/admin.conf
kubectl describe clusterrole system:kubelet-api-admin

PolicyRule:
  Resources   Non-Resource URLs  Resource Names  Verbs
  nodes/proxy []                 []              [*]
  nodes/log   []                 []              [*]
  nodes/metrics []               []              [*]
  nodes/pods  []                 []              [*]
  nodes/stats []                 []              [*]
  nodes       []                 []              [get list watch proxy]

Subresources to hunt in custom Roles and ClusterRoleBindings:

nodes/proxy
nodes/log
nodes/metrics
nodes/stats

Check whether low-privilege identities can use them:

kubectl auth can-i get nodes/proxy --as=system:serviceaccount:default:default

no

The default ServiceAccount in default cannot proxy to nodes in this cluster — a good sign. Cluster-admin can:

kubectl auth can-i '*' nodes/proxy

yes

Review bindings to system:kubelet-api-admin and any custom monitoring Roles during access reviews — see Kubernetes RBAC for least-privilege patterns.

Test kubelet access paths

Record HTTP status and which gate applied. Summary from this lab:

Exercise the API proxy path as cluster-admin:

kubectl get --raw "/api/v1/nodes/worker01/proxy/healthz"

ok

The same identity can reach kubelet paths such as /proxy/pods — powerful during incident response, dangerous when granted to application ServiceAccounts.

From an ordinary Pod on the overlay network, expect either refusal (401/403) or a timeout if routing or firewall blocks Pod CIDR to the node port. Validate on your CNI — a timeout still means the port should not be reachable from workloads.

Explain cloud metadata risk

Cloud and some on-prem platforms expose an instance metadata service (IMDS) at a link-local address. Responses may include:

• Instance identity and region
• Short-lived cloud API credentials
• User-data bootstrap scripts
• Network configuration

A compromised application Pod can query metadata even when Kubernetes RBAC is tight. hostNetwork: true Pods use the node network namespace and inherit the same metadata path as the host unless you add restrictions.

This bare-metal lab has no cloud metadata endpoint. From the worker node:

curl -sk --max-time 3 http://169.254.169.254/latest/meta-data/ -o /dev/null -w "IMDS HTTP %{http_code}\n"

IMDS HTTP 000

On AWS, GCP, or Azure you would see 200 responses with instance metadata unless hop limits or firewall rules block access.

Restrict metadata access

Provider controls differ; combine platform settings with node-level blocks:

Test three contexts after each change:

1. Ordinary Pod on the overlay network
2. hostNetwork Pod (metadata path matches the node)
3. Required system agents and monitoring that legitimately need metadata or kubelet access

Document exceptions with owner and review date like any kube-bench deviation.

Validate required access

After tightening, confirm legitimate automation still works:

• kubectl get --raw nodes/<name>/proxy/healthz as the monitoring ServiceAccount you actually use
• Kubelet client certificate rotation (rotateCertificates: true in config)
• Node metrics scrapers bound to system:monitoring or a custom least-privilege Role
• Cloud agents that need metadata — only on nodes where that remains policy

Re-run targeted checks from kube-bench section 4 (worker node) after kubelet or firewall edits.

Troubleshooting

What's Next

• Verify Kubernetes Release Binaries and Container Images
• Kubernetes RBAC with Examples
• Kubernetes ServiceAccounts with Examples

References

• kubelet authentication and authorization — upstream kubelet access model
• Node authorization mode — API authorization for kubelet-submitted requests
• Protecting kubelet with NodeRestriction — admission control for node objects
• Kubernetes security overview — platform security context

Summary

Kubelet security is a separate boundary from Kubernetes API RBAC. Harden /var/lib/kubelet/config.yaml so anonymous authentication stays off, webhook authentication and webhook authorization are enabled, and the read-only port is not listening. Then test what you actually get: 401 without credentials, 403 with a low-privilege token, and controlled access through nodes/proxy only for identities that truly need it.

Cloud metadata is the second path. Link-local metadata can hand cloud credentials to any workload that can route to it — especially hostNetwork Pods. Platform hop limits, workload identity, and host firewall rules belong in the same review as kubelet port 10250.

For CKS study, pair this lesson with CIS benchmark scanning on worker nodes and the access-path overview in Kubernetes security architecture. Re-test after every kubelet upgrade or firewall change — defaults drift, and a reachable kubelet port with weak RBAC on nodes/proxy is a common lateral-movement pivot.