Kubernetes SecurityContext Explained with Examples

 

Kubernetes SecurityContext Overview

To enforce policies on the pod level, we can use Kubernetes SecurityContext field in the pod specification. A security context is used to define different privilege and access level control settings for any Pod or Container running inside the Pod.

Here are some of the settings which can be configured as part of Kubernetes SecurityContext field:

Advertisement
  • runAsUser to specify the UID with which each container will run
  • runAsNonRoot flag that will simply prevent starting containers that run as UID 0 or root.
  • runAsGroup The GID to run the entrypoint of the container process
  • supplementalGroups specify the Group (GID) for the first process in each container
  • fsGroup we can specify the Group (GID) for filesystem ownership and new files. This can be applied for entire Pod and not on each container.
  • allowPrivilegeEscalation controls whether any process inside the container can gain more privilege to perform the respective task.
  • readOnlyRootFilesystem will mount the container root file system inside the Pod as read-only by default
  • capabilities controls the different capabilities which can be added using 'add' or disabled using 'drop' keyword for the container
  • Seccomp: Filter a process's system calls.
  • AppArmor: Use program profiles to restrict the capabilities of individual programs.
  • Security Enhanced Linux (SELinux) Objects are assigned security labels.

 

Pre-requisites

Before you start with Kubernetes SecurityContext, you must consider below points:

  • You have an up and running Kubernetes Cluster
  • You will need a Pod Security Policy in place which will be used to provide different types of Kubernetes SecurityContext such as privileges, capabilities etc

 

Using runAsUser with Kubernetes SecurityContext

In this section we will explore the runAsUser field used with Kubernetes SecurityContext. The runAsUser can be applied at Pod Level or at Container Level. Let me demonstrate both these examples

 

Example-1: Define runAsUser for entire Pod

In this section we have a multi container pod where we will define runAsUser parameter under Kubernetes SecurityContext for all the containers running inside the Pod.

apiVersion: v1
kind: Pod
metadata:
  name: pod-as-user-guest
  namespace: test1
spec:
  securityContext:
    runAsUser: 1025
  containers:
  - name: one
    image: golinux-registry:8090/secure-context-img:latest
    command: ["/bin/sleep", "999999"]
  - name: two
    image: golinux-registry:8090/secure-context-img:latest
    command: ["/bin/sleep", "999999"]

We will create this Pod:

~]# kubectl create -f security-context-runasuser-1.yaml 
pod/pod-as-user-guest created

Check the status of the Pod, so both our containers are in Running state:

~]# kubectl get pods -n test1
NAME                READY   STATUS    RESTARTS   AGE
pod-as-user-guest   2/2     Running   0          4s

We can connect to both the containers and verify the default user:

Advertisement
~]# kubectl exec -it pod-as-user-guest -n test1 -c one -- id
uid=1025(user2) gid=1025(user2) groups=1025(user2)

~]# kubectl exec -it pod-as-user-guest -n test1 -c two -- id
uid=1025(user2) gid=1025(user2) groups=1025(user2)

As expected, both the containers are running with the provided user id defined with runAsUser under Pod level Kubernetes SecurityContext.

We will delete this pod:

~]# kubectl delete pod pod-as-user-guest -n test1
pod "pod-as-user-guest" deleted

 

Example-2: Define runAsUser for container

In this section now we will define different user for individual container inside the Kubernetes SecurityContext of the Pod definition file:

apiVersion: v1
kind: Pod
metadata:
  name: pod-as-user-guest
  namespace: test1
spec:
  containers:
  - name: one
    image: golinux-registry:8090/secure-context-img:latest
    command: ["/bin/sleep", "999999"]
    securityContext:
      runAsUser: 1025
  - name: two
    image: golinux-registry:8090/secure-context-img:latest
    command: ["/bin/sleep", "999999"]
    securityContext:
      runAsUser: 1026

Here I have defined runAsUser separately for both the containers inside the Kubernetes SecurityContext so we will use different user for both the containers.

Create this Pod:

~]# kubectl create -f security-context-runasuser-1.yaml 
pod/pod-as-user-guest created

Check the status:

~]# kubectl get pods -n test1
NAME                READY   STATUS    RESTARTS   AGE
pod-as-user-guest   2/2     Running   0          94s

Verify the USER ID of both the containers:

# kubectl exec -it pod-as-user-guest -n test1 -c one -- id
uid=1025(user2) gid=1025(user2) groups=1025(user2)

~]# kubectl exec -it pod-as-user-guest -n test1 -c two -- id
uid=1026(user1) gid=1026(user1) groups=1026(user1)

 

Define common group of shared volumes in Kubernetes (fsGroup)

When we are sharing some volumes across multiple containers, then access permission can become a concern. In such scenarios we can use fsGroup under Kubernetes SecurityContext to define a common group which will act as an group owner for any such shared volumes.

Advertisement
NOTE:
fsGroup is assigned at Pod level so you cannot assign it at container level Kubernetes SecurityContext, if you try to assign it at container level then you will get below error:
error: error validating "security-context-fsgroup-1.yaml": error validating data: [ValidationError(Pod.spec.containers[0].securityContext): unknown field "fsGroup" in io.k8s.api.core.v1.SecurityContext, ValidationError(Pod.spec.containers[1].securityContext): unknown field "fsGroup" in io.k8s.api.core.v1.SecurityContext]; if you choose to ignore these errors, turn validation off with --validate=false

This our sample YAML file to create a pod using fsGroup:

apiVersion: v1
kind: Pod
metadata:
  name: pod-as-user-guest
  namespace: test1
spec:
  securityContext:
    fsGroup: 555
  containers:
  # create container one
  - name: one
    image: golinux-registry:8090/secure-context-img:latest
    command: ["/bin/sleep", "999999"]
    # container one running as user id 1025
    securityContext:
      runAsUser: 1025
    # mount empty dir under /volume
    volumeMounts:
    - name: shared-volume
      mountPath: /volume
  # create container two
  - name: two
    image: golinux-registry:8090/secure-context-img:latest
    command: ["/bin/sleep", "999999"]
   # container one running as user id 1026
    securityContext:
      runAsUser: 1026
    # mount empty dir under /volume
    volumeMounts:
    - name: shared-volume
      mountPath: /volume
  # create emptyDir
  volumes:
  - name: shared-volume
    emptyDir: {}

Create this pod:

~]# kubectl create -f security-context-fsgroup-1.yaml 
pod/pod-as-user-guest created

Verify the group ownership on the shared volume:

~]# kubectl exec -it pod-as-user-guest -n test1 -c one -- bash
[user2@pod-as-user-guest /]$ id
uid=1025(user2) gid=1025(user2) groups=1025(user2),555

[user2@pod-as-user-guest /]$ ls -ld /volume/
drwxrwsrwx. 2 root 555 4096 Sep  3 09:28 /volume/

[user2@pod-as-user-guest /]$ exit

So, one container one the /volume path is owned by 555 GID as expected. The id command shows the container is running with user ID 1025, as specified in the pod definition. The effective group ID is 1025(user2), but group ID 555 is also associated with the user.

Let's verify the same on two container:

~]# kubectl exec -it pod-as-user-guest -n test1 -c two -- bash
[user1@pod-as-user-guest /]$ id
uid=1026(user1) gid=1026(user1) groups=1026(user1),555

[user1@pod-as-user-guest /]]$ ls -ld /volume/
drwxrwsrwx. 2 root 555 4096 Sep  3 09:28 /volume/

One more thing which you should know that with fsGroup Kubernetes SecurityContext, any files created inside the shared volume will have group ownership of the ID provided in the pod definition file. For example, here I will create a file inside /tmp:

Advertisement
[user1@pod-as-user-guest ~]$ touch /tmp/file

[user1@pod-as-user-guest ~]$ ls -l /tmp/file
-rw-rw-r--. 1 user1 user1 0 Sep  3 09:42 /tmp/file

As you can see above, the file is owned by user2 user and group. But if you create any file inside the shared volume i.e. /volume path then the group owner of that respective file would be same as fsGroup value i.e. 555 in our case:

[user1@pod-as-user-guest ~]$ touch /volume/file

[user1@pod-as-user-guest ~]$ ls -l /volume/file 
-rw-rw-r--. 1 user1 555 0 Sep  3 09:42 /volume/file

As you can see, the fsGroup Kubernetes SecurityContext property is used when the process creates files in a volume (but this depends on the volume plugin used).

 

Define supplementalGroups inside Kubernetes SecurityContext

We can combine fsGroup with supplementalGroups inside the Pod's SecurityContext field to define some additional groups. In such case the runAsUser or the default image user will also be added to these supplementary groups.

apiVersion: v1
kind: Pod
metadata:
  name: pod-as-user-guest
  namespace: test1
spec:
  securityContext:
    fsGroup: 555
    # Define additional groups for the default user
    supplementalGroups: [666, 777]
  containers:
  # create container one
  - name: one
    image: golinux-registry:8090/secure-context-img:latest
    command: ["/bin/sleep", "999999"]
...

We will create this Pod and verify the list of groups part of user2 user:

~]# kubectl exec -it pod-as-user-guest -n test1 -c two -- id
uid=1026(user1) gid=1026(user1) groups=1026(user1),555,666,777

So now along with fsGroup, our user has also been added to additional supplementary groups.

 

Using allowPrivilegeEscalation with Kubernetes SecurityContext

In this section we will cover different areas related to privilege where we will add or remove capabilities to the container. Every container inside a Pod uses kernel capabilities to perform different task, such as even when you are changing permission of a file then you can basically using CAP_CHOWN capability and many other capability such as CAP_SUID, CAP_SGID etc. You use CAP_SYS_MOUNT capability when you are using mount, umount commands.

So, with containers you can accordingly assign or drop these capabilities so the user inside the container will have limited privilege and is considered more secure. To be able to control or restrict capabilities, you must define allowPrivilegeEscalation as true inside the Pod's Kubernetes Security Context.

The capabilities which a pod would use are basically defined using PodSecurityPolicy.

 

Example-1: Using allowedCapabilities in Pod Security Policy

The allowedCapabilities field is used to specify which capabilities pod authors can add in the Kubernetes securityContext.capabilities field in the container spec.

I have following PSP currently added to my Kubernetes Cluster and I have added some capabilities under allowedCapabilities:

~]# kubectl get psp testns-psp-01 -o yaml
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
...
spec:
  allowPrivilegeEscalation: true
  allowedCapabilities:
  - SYS_ADMIN
  - NET_BIND_SERVICE
  - CHOWN
  requiredDropCapabilities:
  - ALL
...

Status of the PSP can be checked using below command:

~]# kubectl get psp | grep -E 'PRIV|testns'
NAME                                PRIV    CAPS                               SELINUX    RUNASUSER          FSGROUP     SUPGROUP    READONLYROOTFS   VOLUMES
testns-psp-01                       false   SYS_ADMIN,NET_BIND_SERVICE,CHOWN   RunAsAny   MustRunAsNonRoot   RunAsAny    RunAsAny    false            *

So we are only allowing capabilities as mentioned under CAPS section.

We will create a StatefulSet with certain pre-defined capabilities, but the capability we use will not be part of allowed capabilities in the Pod Security Policy. Here is my pod definition file:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: test-statefulset
  namespace: testns
spec:
  selector:
    matchLabels:
      app: dev
  serviceName: test-pod
  replicas: 2
  template:
    metadata:
      labels:
        app: dev
    spec:
      containers:
      - name: test-statefulset
        image: golinux-registry:8090/secure-context-img:latest
        command: ["supervisord", "-c", "/etc/supervisord.conf"]
        imagePullPolicy: Always
        securityContext:
          runAsUser: 1025
          ## by default the privilege will be disabled
          privileged: false
          ## allow the use of capabilitites
          allowPrivilegeEscalation: true
          capabilities:
            ## drop all capabilities
            drop:
             - ALL
            ## The creation of statefulset should fail as SUID is not in allowedCapabilities
            add:
             - SUID

Here, we are trying to use a capability using Kubernetes SecurityContext which has not been defined in Pod Security Policy, so let's try to create this statefulset:

~]# kubectl create -f test-statefulset.yaml 
statefulset.apps/test-statefulset created

The statefulset has been successfully created but the pods have not come up:

~]# kubectl get statefulset -n testns
NAME                      READY   AGE
test-statefulset          0/2     12m

We can use kubectl describe statefulset test-statefulset -n testns command to troubleshoot the issue here:

Events:
  Type     Reason        Age                From                    Message
  ----     ------        ----               ----                    -------
  Warning  FailedCreate  10s (x6 over 12s)  statefulset-controller  create Pod test-statefulset-0 in StatefulSet test-statefulset failed error: pods "test-statefulset-0" is forbidden: unable to validate against any pod security policy: [spec.containers[0].securityContext.capabilities.add: Invalid value: "SUID": capability may not be added spec.containers[0].securityContext.allowPrivilegeEscalation: Invalid value: true: Allowing privilege escalation for containers is not allowed spec.containers[0].securityContext.capabilities.add: Invalid value: "SUID": capability may not be added spec.containers[0].securityContext.capabilities.add: Invalid value: "SUID": capability may not be added]
  Warning  FailedCreate  7s (x2 over 12s)   statefulset-controller  create Pod test-statefulset-0 in StatefulSet test-statefulset failed error: pods "test-statefulset-0" is forbidden: unable to validate against any pod security policy: [spec.containers[0].securityContext.capabilities.add: Invalid value: "SUID": capability may not be added spec.containers[0].securityContext.capabilities.add: Invalid value: "SUID": capability may not be added spec.containers[0].securityContext.capabilities.add: Invalid value: "SUID": capability may not be added spec.containers[0].securityContext.allowPrivilegeEscalation: Invalid value: true: Allowing privilege escalation for containers is not allowed]
  Warning  FailedCreate  2s (x4 over 12s)   statefulset-controller  create Pod test-statefulset-0 in StatefulSet test-statefulset failed error: pods "test-statefulset-0" is forbidden: unable to validate against any pod security policy: [spec.containers[0].securityContext.capabilities.add: Invalid value: "SUID": capability may not be added spec.containers[0].securityContext.capabilities.add: Invalid value: "SUID": capability may not be added spec.containers[0].securityContext.allowPrivilegeEscalation: Invalid value: true: Allowing privilege escalation for containers is not allowed spec.containers[0].securityContext.capabilities.add: Invalid value: "SUID": capability may not be added]

As expected, since the SUID capability was not defined in the PodSecurityPolicy so statefulset failed to create pods.

Now I have modified my statefulset definition file and use CHOWN capability instead of SUID and the pod creation was successful. We can verify the same using the following command:

~]# kubectl exec -it test-statefulset-0 -n testns -- capsh --print
Current: = cap_chown+i
Bounding set =cap_chown
Securebits: 00/0x0/1'b0
 secure-noroot: no (unlocked)
 secure-no-suid-fixup: no (unlocked)
 secure-keep-caps: no (unlocked)
uid=1025(user2)
gid=1025(user2)
groups=

 

Example-2: Using defaultAddCapabilities in PodSecurityPolicy

Next we will update our Pod Security Policy to also add some defaultAddCapabilities. All capabilities listed under the defaultAddCapabilities field will be added to every deployed pod’s containers. If a user doesn’t want certain containers to have those capabilities, they need to explicitly drop them in the specs of those containers.

I have modified my testns-psp-01 using kubectl edit psp testns-psp-01 -n testns command and added defaultAddCapabilities field with new capability:

...
spec:
  allowPrivilegeEscalation: true
  allowedCapabilities:
  - SYS_ADMIN
  - NET_BIND_SERVICE
  - CHOWN
  defaultAddCapabilities:
  - NET_RAW
  fsGroup:
    rule: RunAsAny
  requiredDropCapabilities:
  - ALL
...

So, we have marked NET_RAW as default capability which will be added to any container using this Pod Security Policy.

Let us quickly create one statefulset using our last pod definition file and verify if NET_RAW capability is automatically added to the container:

...
        securityContext:
          runAsUser: 1025
          ## by default the privilege will be disabled
          privileged: false
          ## allow the use of capabilitites
          allowPrivilegeEscalation: true
          capabilities:
            ## drop all capabilities
            drop:
             - ALL
            ## Add chown capability
            add:
             - CHOWN
...

Let's create this statefulset and verify the list of allowed capabilities:

~]# kubectl exec -it test-statefulset-0 -n testns -- capsh --print
Current: = cap_chown,cap_net_raw+i
Bounding set =cap_chown,cap_net_raw
Securebits: 00/0x0/1'b0
 secure-noroot: no (unlocked)
 secure-no-suid-fixup: no (unlocked)
 secure-keep-caps: no (unlocked)
uid=1025(user2)
gid=1025(user2)
groups=

As expected, we had only added CHOWN capability but our pod also contains NET_RAW capability which was added as part of defaultAddCapabilities from Pod Security Policy.

 

Example-3: Using requiredDropCapabilities in Pod Security Policy

The final field in this example is requiredDropCapabilities. The capabilities listed in this field are dropped automatically from every container (the PodSecurityPolicy Admission Control plugin will add them to every container’s securityContext.capabilities.drop field).

We have updated our ststefulset definition file, and now we are not dropping or adding any additional capability:

        ...
        securityContext:
          runAsUser: 1025
          ## by default the privilege will be disabled
          privileged: false
          ## allow the use of capabilitites
          allowPrivilegeEscalation: true
          ...

We have also updated our PodSecurityPolicy using kubectl edit psp testns-psp-01 -n testns and added SYS_ADMIN as requiredDropCapabilities:

....
spec:
  allowPrivilegeEscalation: true
  allowedCapabilities:
  - NET_BIND_SERVICE
  - CHOWN
  defaultAddCapabilities:
  - NET_RAW
  fsGroup:
    rule: RunAsAny
  requiredDropCapabilities:
  - SYS_ADMIN
....

Next, we deploy our statefulset and verify the applied Linux capabilities:

~]# kubectl exec -it test-statefulset-0 -n testns -- capsh --print
Current: = cap_chown,cap_dac_override,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_net_bind_service,cap_net_raw,cap_sys_chroot,cap_mknod,cap_audit_write,cap_setfcap+i
Bounding set =cap_chown,cap_dac_override,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_net_bind_service,cap_net_raw,cap_sys_chroot,cap_mknod,cap_audit_write,cap_setfcap
Securebits: 00/0x0/1'b0
 secure-noroot: no (unlocked)
 secure-no-suid-fixup: no (unlocked)
 secure-keep-caps: no (unlocked)
uid=1025(user2)
gid=1025(user2)
groups=

Here you can check, SYS_ADMIN capability is not available as it is removed using requiredDropCapabilities

 

What would happen if you explicitly try to add a dropped capability in SecurityContext?

Here we have dropped SYS_ADMIN capability using requiredDropCapabilities in the Pod Security Policy. Now what if we explicitly try to add the same capability to our statefulset using Kubernetes SecurityContext field as shown below:

...
        securityContext:
          runAsUser: 1025
          ## by default the privilege will be disabled
          privileged: false
          ## allow the use of capabilitites
          allowPrivilegeEscalation: true
          capabilities:
            add:
             - SYS_ADMIN
...

Next, lets try to create this statefulset:

~]# kubectl create -f test-statefulset.yaml 
statefulset.apps/test-statefulset created

Now let's check the events of this statefulset for any errors:

~]# kubectl describe statefulset test-statefulset -n testns
...
Events:
  Type     Reason        Age              From                    Message
  ----     ------        ----             ----                    -------
  Warning  FailedCreate  5s               statefulset-controller  create Pod test-statefulset-0 in StatefulSet test-statefulset failed error: pods "test-statefulset-0" is forbidden: unable to validate against any pod security policy: [spec.containers[0].securityContext.capabilities.add: Invalid value: "SYS_ADMIN": capability may not be added spec.containers[0].securityContext.allowPrivilegeEscalation: Invalid value: true: Allowing privilege escalation for containers is not allowed spec.containers[0].securityContext.capabilities.add: Invalid value: "SYS_ADMIN": capability may not be added spec.containers[0].securityContext.capabilities.add: Invalid value: "SYS_ADMIN": capability may not be added]
  Warning  FailedCreate  5s (x3 over 5s)  statefulset-controller  create Pod test-statefulset-0 in StatefulSet test-statefulset failed error: pods "test-statefulset-0" is forbidden: unable to validate against any pod security policy: [spec.containers[0].securityContext.capabilities.add: Invalid value: "SYS_ADMIN": capability may not be added spec.containers[0].securityContext.capabilities.add: Invalid value: "SYS_ADMIN": capability may not be added spec.containers[0].securityContext.capabilities.add: Invalid value: "SYS_ADMIN": capability may not be added spec.containers[0].securityContext.allowPrivilegeEscalation: Invalid value: true: Allowing privilege escalation for containers is not allowed]
  Warning  FailedCreate  3s (x6 over 5s)  statefulset-controller  create Pod test-statefulset-0 in StatefulSet test-statefulset failed error: pods "test-statefulset-0" is forbidden: unable to validate against any pod security policy: [spec.containers[0].securityContext.capabilities.add: Invalid value: "SYS_ADMIN": capability may not be added spec.containers[0].securityContext.capabilities.add: Invalid value: "SYS_ADMIN": capability may not be added spec.containers[0].securityContext.allowPrivilegeEscalation: Invalid value: true: Allowing privilege escalation for containers is not allowed spec.containers[0].securityContext.capabilities.add: Invalid value: "SYS_ADMIN": capability may not be added]

Hence if a user tries to create a pod where they explicitly add one of the capabilities listed in the policy’s requiredDropCapabilities field, the pod is rejected:

 

Summary

In this tutorial we explored different Kubernetes SecurityContext which we can use inside a Pod and container. In a nutshell we covered following topics:

  • Containers can be configured to run as a different user and/or group than the one defined in the container image.
  • Containers can also run in privileged mode, allowing them to access the node’s devices that are otherwise not exposed to pods.
  • Containers can be run as read-only, preventing processes from writing to the container’s filesystem (and only allowing them to write to mounted volumes).
  • Cluster-level PodSecurityPolicy resources can be created to prevent users from creating pods that could compromise a node.

 

Further Readings

Configure a Security Context for a Pod or Container

 

Didn't find what you were looking for? Perform a quick search across GoLinuxCloud

If my articles on GoLinuxCloud has helped you, kindly consider buying me a coffee as a token of appreciation.

Buy GoLinuxCloud a Coffee

For any other feedbacks or questions you can either use the comments section or contact me form.

Thank You for your support!!

Leave a Comment