Examples to secure Kubernetes cluster nodes (security context)

Advertisement

In the previous chapter, we talked about securing the API server. If an attacker gets access to the API server, they can run whatever they like by packaging their code into a container image and running it in a pod. But can they do any real damage? Aren’t containers isolated from other containers and from the node they’re running on?
Not necessarily. In this chapter, you’ll learn how to allow pods to access the resources of the node they’re running on.

 

Using the host node’s namespace in a pod

Containers in a pod usually run under separate Linux namespaces, which isolate their processes from processes running in other containers or in the node’s default namespaces.

For example, we learned that each pod gets its own IP and port space, because it uses its own network namespace. Likewise, each pod has its own process tree, because it has its own PID namespace, and it also uses its own IPC namespace, allowing only processes in the same pod to communicate with each other through the Inter-Process Communication mechanism (IPC).

 

Using the node’s network namespace in a pod

Certain pods (usually system pods) need to operate in the host’s default namespaces, allowing them to see and manipulate node-level resources and devices. For example, a pod may need to use the node’s network adapters instead of its own virtual network adapters. This can be achieved by setting the hostNetwork property in the pod spec to true.

Here, we are creating one such pod with hostNetwork: true as spec properties:

[root@controller ~]# cat pod-with-host-network.yml
apiVersion: v1
kind: Pod
metadata:
  name: pod-with-host-network
spec:
  hostNetwork: true
  containers:
  - name: main
    image: alpine
    command: ["/bin/sleep", "999999"]

Next create the pod:

[root@controller ~]# kubectl create -f pod-with-host-network.yml
pod/pod-with-host-network created

And verify the status of the pod

Advertisement
[root@controller ~]# kubectl get pods
NAME                    READY   STATUS    RESTARTS   AGE
pod-with-host-network   1/1     Running   0          64s
sidecar-pod             2/2     Running   2          11h

Now we can connect to the container from this new pod and check the list of available interfaces:

[root@controller ~]# kubectl exec pod-with-host-network -- ifconfig
datapath: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1376
        ether 22:a9:66:98:e7:f3  txqueuelen 1000  (Ethernet)
        RX packets 31  bytes 1624 (1.5 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

docker0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 172.17.0.1  netmask 255.255.0.0  broadcast 172.17.255.255
        ether 02:42:29:35:5d:9b  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.0.2.15  netmask 255.255.255.0  broadcast 10.0.2.255
        ether 08:00:27:b6:89:b5  txqueuelen 1000  (Ethernet)
        RX packets 711  bytes 61270 (59.8 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 395  bytes 67922 (66.3 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
...

As expected, now the interfaces from host node is visible inside the container.

 

Binding to a host port without using the host’s network namespace

A related feature allows pods to bind to a port in the node’s default namespace, but still have their own network namespace. This is done by using the hostPort property in one of the container’s ports defined in the spec.containers.ports field. The hostPort feature is primarily used for exposing system services, which are deployed to every node using DaemonSets.

It’s important to understand that if a pod is using a specific host port, only one instance of the pod can be scheduled to each node, because two processes can’t bind to the same host port.

Here I have created a nginx container with hostPort so we can access the container from external network from the same host where container is running:

[root@controller ~]# cat nginx-lab.yml
apiVersion: v1
kind: Pod
metadata:
  name: nginx-lab
  namespace: default
spec:
  containers:
  - name: nginx-lab
    image: nginx
    ports:
    - containerPort: 80
      hostPort: 9000
      protocol: TCP

With port 80 the container can be reached on port 8080 of the pod’s IP, while it can also be reached on port 9000 of the node it’s deployed on.

Create the pod using this YAML file:

[root@controller ~]# kubectl create -f nginx-lab.yml
pod/nginx-lab created

Next verify the status of the newly created pod to make sure it is in Running state:

Advertisement
[root@controller ~]# kubectl get pods -o wide
[root@controller ~]# kubectl create -f nginx-lab.yml
pod/nginx-lab created
[root@controller ~]# kubectl get pods -o wide
NAME          READY   STATUS    RESTARTS   AGE     IP          NODE                   NOMINATED NODE   READINESS GATES
nginx         1/1     Running   0          7m11s   10.44.0.1   worker-2.example.com   <none>           <none>
nginx-lab     1/1     Running   0          16s     10.44.0.2   worker-2.example.com   <none>           <none>
sidecar-pod   2/2     Running   2          11h     10.36.0.1   worker-1.example.com   <none>           <none>

Now we can access our container from external network using hostPort i.e. 9000:

[root@worker-2 ~]# curl 127.0.0.1:9000
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

 

Using the node’s PID and IPC namespaces

Similar to the hostNetwork option are the hostPID and hostIPC pod spec properties. When you set them to true, the pod’s containers will use the node’s PID and IPC namespaces, allowing processes running in the containers to see all the other processes on the node or communicate with them through IPC, respectively.

Here we will create another pod using alpine image to demonstrate the behaviour of hostPID and hostIPC:

[root@controller ~]# cat pod-with-host-pid-and-ipc.yml
apiVersion: v1
kind: Pod
metadata:
  name: pod-with-host-pid-and-ipc
spec:
  hostPID: true
  hostIPC: true
  containers:
  - name: main
    image: alpine
    command: ["/bin/sleep", "999999"]

If you run this pod and then list the processes from within its container, you’ll see all the processes running on the host node, not only the ones running in the container, as shown in the following listing.

[root@controller ~]# kubectl exec pod-with-host-pid-and-ipc -- ps aux
PID   USER     TIME  COMMAND
    1 root      0:03 /usr/lib/systemd/systemd --switched-root --system --deserialize 18
    2 root      0:00 [kthreadd]
    3 root      0:00 [rcu_gp]
    4 root      0:00 [rcu_par_gp]
    6 root      0:00 [kworker/0:0H-kb]
    8 root      0:00 [mm_percpu_wq]
    9 root      0:00 [ksoftirqd/0]
   10 root      0:02 [rcu_sched]
   11 root      0:00 [migration/0]
   12 root      0:00 [watchdog/0]
   13 root      0:00 [cpuhp/0]
   14 root      0:00 [cpuhp/1]
   15 root      0:00 [watchdog/1]
   16 root      0:00 [migration/1]
...

 

Configure container’s security context

Besides allowing the pod to use the host’s Linux namespaces, other security-related features can also be configured on the pod and its container through the security-Context properties, which can be specified under the pod spec directly and inside the spec of individual containers.

Configuring the security context allows you to do various things:

  • Specify the user (the user’s ID) under which the process in the container will run.
  • Prevent the container from running as root (the default user a container runs as is usually defined in the container image itself, so you may want to prevent containers from running as root).
  • Run the container in privileged mode, giving it full access to the node’s kernel.
  • Configure fine-grained privileges, by adding or dropping capabilities—in contrast to giving the container all possible permissions by running it in privileged mode.
  • Set SELinux (Security Enhanced Linux) options to strongly lock down a container.
  • Prevent the process from writing to the container’s filesystem.

 

Running a pod without specifying a security context

First, run a pod with the default security context options (by not specifying them at all), so you can see how it behaves compared to pods with a custom security context:

[root@controller ~]# kubectl run pod-with-defaults --image alpine --restart Never -- /bin/sleep 999999
pod/pod-with-defaults created

Once the container inside this Pod is created, let’s see what user and group ID the container is running as, and which groups it belongs to. You can see this by running the id command inside the container:

[root@controller ~]# kubectl exec pod-with-defaults -- id
uid=0(root) gid=0(root) groups=0(root),1(bin),2(daemon),3(sys),4(adm),6(disk),10(wheel),11(floppy),20(dialout),26(tape),27(video)

The container is running as user ID (uid) 0, which is root, and group ID (gid) 0 (also root). It’s also a member of multiple other groups.

 

Running a container as a specific user

To run a pod under a different user ID than the one that’s baked into the container image, you’ll need to set the pod’s securityContext.runAsUser property. You’ll make the container run as user guest, whose user ID in the alpine container image is 1003, as shown in the following listing.

Advertisement
[root@controller ~]# cat pod-as-user-guest.yml
apiVersion: v1
kind: Pod
metadata:
  name: pod-as-user-guest
spec:
  containers:
  - name: main
    image: alpine
    command: ["/bin/sleep", "999999"]
    securityContext:
      runAsUser: 405

You must specify a user ID, not a username (id 405 corresponds to the guest user).

Next create the pod and verify the status:

[root@controller ~]# kubectl create -f pod-as-user-guest.yml
pod/pod-as-user-guest created

[root@controller ~]# kubectl get pods -o wide
NAME                        READY   STATUS              RESTARTS   AGE     IP          NODE                   NOMINATED NODE   READINESS GATES
nginx                       1/1     Running             0          44m     10.44.0.1   worker-2.example.com   <none>           <none>
nginx-lab                   1/1     Running             0          37m     10.44.0.2   worker-2.example.com   <none>           <none>
pod-as-user-guest           0/1     ContainerCreating   0          3s      <none>      worker-2.example.com   <none>           <none>
pod-with-defaults           1/1     Running             0          6m16s   10.36.0.2   worker-1.example.com   <none>           <none>
pod-with-host-pid-and-ipc   1/1     Running             0          15m     10.36.0.1   worker-1.example.com   <none>           <none>

Once the status changes from ContainerCreating to Running, you can execute the following command to verify the user:

[root@controller ~]# kubectl exec pod-as-user-guest -- id
uid=1003 gid=1000(users)

As requested, the container is running as the guest user.

 

Preventing a container from running as root

What if you don’t care what user the container runs as, but you still want to prevent it from running as root? To prevent the attack scenario described previously, you can specify that the pod’s container needs to run as a non-root user, as shown in the following listing.

[root@controller ~]# cat pod-run-as-non-root.yml
apiVersion: v1
kind: Pod
metadata:
  name: pod-run-as-non-root
spec:
  containers:
  - name: main
    image: alpine
    command: ["/bin/sleep", "999999"]
    securityContext:
      runAsNonRoot: true

Next create the pod and verify the status:

[root@controller ~]# kubectl create -f pod-run-as-non-root.yml
pod/pod-run-as-non-root created

As you can check the status, we get CreateContainerConfigError

[root@controller ~]# kubectl get po pod-run-as-non-root
NAME                  READY   STATUS                       RESTARTS   AGE
pod-run-as-non-root   0/1     CreateContainerConfigError   0          2m1s

We can get more details on this using describe (output is snipped):

[root@controller ~]# kubectl describe pods pod-run-as-non-root
..
  Warning  Failed     54s (x8 over 2m58s)  kubelet            Error: container has runAsNonRoot and image will run as root
..

As expected, the container creation failed as we were trying to create it as root user:

 

Running pods in privileged mode

Sometimes pods need to do everything that the node they’re running on can do, such as use protected system devices or other kernel features, which aren’t accessible to regular containers. To get full access to the node’s kernel, the pod’s container runs in privileged mode. This is achieved by setting the privileged property in the container’s security-Context property to true

[root@controller ~]# cat pod-privileged.yml
apiVersion: v1
kind: Pod
metadata:
  name: pod-privileged
spec:
  containers:
  - name: main
    image: alpine
    command: ["/bin/sleep", "999999"]
    securityContext:
      privileged: true

Next create this pod and verify the status is Running:

[root@controller ~]# kubectl create -f pod-privileged.yml
pod/pod-privileged created

If you’re familiar with Linux, you may know it has a special file directory called /dev, which contains device files for all the devices on the system. These aren’t regular files on disk, but are special files used to communicate with devices. Let’s see what devices are visible in the non-privileged container you deployed earlier (the pod-with-defaults pod), by listing files in its /dev directory

[root@controller ~]# kubectl exec -it pod-with-defaults -- ls /dev
core             null             shm              termination-log
fd               ptmx             stderr           tty
full             pts              stdin            urandom
mqueue           random           stdout           zero

The listing shows all the devices. The list is fairly short. Now, compare this with the following listing, which shows the device files your privileged pod can see

[root@controller ~]# kubectl exec -it pod-privileged -- ls /dev
autofs              stderr              tty5
bsg                 stdin               tty50
bus                 stdout              tty51
core                termination-log     tty52
cpu                 tty                 tty53
cpu_dma_latency     tty0                tty54
dm-0                tty1                tty55
dm-1                tty10               tty56
dri                 tty11               tty57
fb0                 tty12               tty58
...                 ...                 ...

 

Adding individual kernel capabilities to a container

Instead of making a container privileged and giving it unlimited permissions, a much safer method (from a security perspective) is to give it access only to the kernel features it really requires. Kubernetes allows you to add capabilities to each container or drop part of them, which allows you to fine-tune the container’s permissions and limit the impact of a potential intrusion by an attacker.

For example, a container usually isn’t allowed to change the system time (the hardware clock’s time). You can confirm this by trying to set the time in your pod-with-defaults pod:

[root@controller ~]# kubectl exec -it pod-with-defaults -- date +%T -s "12:00:00"
date: can't set date: Operation not permitted
12:00:00

If you want to allow the container to change the system time, you can add a capability called CAP_SYS_TIME to the container’s capabilities list, as shown in the following listing.

[root@controller ~]# cat pod-add-settime-capability.yml
apiVersion: v1
kind: Pod
metadata:
  name: pod-add-settime-capability
spec:
  containers:
  - name: main
    image: alpine
    command: ["/bin/sleep", "999999"]
    securityContext:
      capabilities:
        add:
        - SYS_TIME

Now create this pod and attempt to change the date once the container is created:

[root@controller ~]# kubectl create -f pod-add-settime-capability.yml
pod/pod-add-settime-capability created

[root@controller ~]#  kubectl exec -it pod-add-settime-capability -- date +%T -s "12:00:00"; date
12:00:00

[root@controller ~]#  kubectl exec -it pod-add-settime-capability -- date
Sat Nov 28 12:00:20 UTC 2020

You can confirm the node’s time has been changed by checking the time on the node running the pod.

 

Dropping capabilities from a container

You can also drop capabilities that may otherwise be available to the container. For example, the default capabilities given to a container include the CAP_CHOWN capability, which allows processes to change the ownership of files in the filesystem.

You can see that’s the case by changing the ownership of the /tmp directory in your pod-with-defaults pod to the guest user, for example:

[root@controller ~]# kubectl exec pod-with-defaults -- chown guest /tmp

[root@controller ~]# kubectl exec pod-with-defaults -- ls -la / | grep tmp
drwxrwxrwt    1 guest    root          4096 Oct 21 09:23 tmp

To prevent the container from doing that, you need to drop the capability by listing it under the container’s securityContext.capabilities.drop property, as shown in the following listing.

[root@controller ~]# cat pod-drop-chown-capability.yml
apiVersion: v1
kind: Pod
metadata:
  name: pod-drop-chown-capability
spec:
  containers:
  - name: main
    image: alpine
    command: ["/bin/sleep", "999999"]
    securityContext:
      capabilities:
        drop:
        - CHOWN

By dropping the CHOWN capability, you’re not allowed to change the owner of the /tmp directory in this pod:

[root@controller ~]# kubectl exec pod-drop-chown-capability -- chown guest /tmp
chown: /tmp: Operation not permitted
command terminated with exit code 1

 

Preventing processes from writing to the container’s filesystem

You may want to prevent the processes running in the container from writing to the container’s filesystem, and only allow them to write to mounted volumes. This is done by setting the container’s securityContext.readOnlyRootFilesystem property to true, as shown in the following listing.

[root@controller ~]# cat pod-with-readonly-filesystem.yml
apiVersion: v1
kind: Pod
metadata:
  name: pod-with-readonly-filesystem
spec:
  containers:
  - name: main
    image: alpine
    command: ["/bin/sleep", "999999"]
    securityContext:
      readOnlyRootFilesystem: true
    volumeMounts:
    - name: my-volume
      mountPath: /volume
      readOnly: false
  volumes:
  - name: my-volume
    emptyDir:

In this container’s filesystem can’t be written to but writing to /volume is allowed, becase a volume is mounted there.

Next create this pod and make sure the containers are created before we do any verification:

[root@controller ~]# kubectl create -f pod-with-readonly-filesystem.yml
pod/pod-with-readonly-filesystem created

When you deploy this pod, the container is running as root, which has write permissions to the / directory, but trying to write a file there fails:

[root@controller ~]# kubectl exec -it pod-with-readonly-filesystem -- touch /new-file
touch: /new-file: Read-only file system

On the other hand, writing to the mounted volume is allowed:

[root@controller ~]# kubectl exec -it pod-with-readonly-filesystem -- touch /volume/newfile

[root@controller ~]# kubectl exec -it pod-with-readonly-filesystem -- ls -la /volume/newfile
-rw-r--r--    1 root     root       0 Nov  28 19:11 /mountedVolume/newfile

 

Conclusion

In this Kubernetes Tutorial we covered different areas of securing cluster nodes. This is really a vast topic and contains many more areas which couldnot be covered in this tutorial, I may write another tutorial covering the remaining part of securing the cluster nodes such as RBAC ClusterRoles, ClusterRoleBindings, NetworkPolicy resources that are used to limit a pod’s inbound and/or outbound traffic. Following are the topics covered in this tutorial:

  • Containers can be configured to run as a different user and/or group than the one defined in the container image.
  • Containers can also run in privileged mode, allowing them to access the node’s devices that are otherwise not exposed to pods.
  • Containers can be run as read-only, preventing processes from writing to the container’s filesystem (and only allowing them to write to mounted volumes).

If my articles on GoLinuxCloud has helped you, kindly consider buying me a coffee as a token of appreciation.

Buy GoLinuxCloud a Coffee

For any other feedbacks or questions you can either use the comments section or contact me form.

Thank You for your support!!

Leave a Comment