Kubernetes Secrets | Declare confidential data with examples

In this article we will explore the usage of Kubernetes Secrets using some real time scenarios and examples. Ideally we use Secrets to declare confidential and secret data to the Pod such as username or password.

When should you choose Secrets over ConfigMaps in Kubernetes?

You may wonder why should I use secrets to declare the environment variables over ConfigMaps in Kubernetes. You may very well use either unless you are worried about security. Because with configmap the data is declared or transferred to Pod in plain text format while with Kubernetes Secrets the data will be sent in base64 encoded format.

So if you have a requirement to set a password environment variable then you should use Secrets for this purpose but if you have some non-confidential data such as server name, path etc then you can use ConfigMap which can be safely transferred in plain text format.

Using Kubernetes Secrets to pass sensitive data to containers

• While ConfigMaps are great for most configuration data, there is certain data that is extra-sensitive. This can include passwords, security tokens, or other types of private keys. Collectively, we call this type of data “secrets.”
• Secrets enable container images to be created without bundling sensitive data. This allows containers to remain portable across environments.
• Unlike a ConfigMap, Kubernetes Secrets are intended to store a small amount (1 MB for a Secret) of sensitive data.
• A Secret is base64-encoded, so we cannot treat it as secure.
• It can also store binary data such as a public or private key.
• Kubernetes ensures that Secrets are passed only to the nodes that are running the Pods that need the respective Secrets.

How to create Kubernetes Secrets?

Secrets can be created through

• a literal value
• from a file or all the files in a directory
• Manually defining the key value in YAML file format

we will explore all these options in this tutorial.

The command syntax to create a Secret has the following format:

kubectl create secret <map-name> <data-source>

Here, is the name you want to assign to the Secret and is the directory, file, or literal value to draw the data from. The corresponds to a key-value pair in the Secret, where:

• Key is the filename or the key you provided on the command line
• Value is the file content or the literal value you provided on the command line

When creating Secrets, you can use a combination of all the options mentioned here:

$ kubectl create secret my-secret
➥   --from-file=foo.json         --> A single file
➥   --from-file=bar=foobar.conf  --> A file stored under a custom key
➥   --from-file=config-opts/     --> A whole directory
➥   --from-literal=some=thing    --> A literal value

Method-1: Declare Kubernetes Secrets as Environment Variables

In this section we will declare Kubernetes Secrets as Environment variables in the Pod. Now there are multiple ways to define kubernetes secrets, so let's explore some of them using practical examples:

Example-1: Declare Kubernetes Secret from Literal Values

In this example, we will define a Secret from a literal value and load it as secrets volume type. Secrets volumes are managed by the kubelet and are created at Pod creation time. Secrets are stored on tmpfs volumes (aka RAM disks), and as such are not written to disk on nodes. Each data element of a secret is stored in a separate file under the target mount point specified in the volume mount.

Create Kubernetes Secret using literal values

Now this literal value maybe something like a password to your internal database. Since we are creating this Secret from a literal value, it would be categorized as a generic Secret:

[root@controller ~]# kubectl create secret generic db-secret --from-literal=db_user=deepak --from-literal=db_password=test1234
secret/db-secret created

Once we define our Secret, we can use the Kubernetes get command to obtain more details about it:

[root@controller ~]# kubectl get secret db-secret -o yaml
apiVersion: v1
data:
  db_password: dGVzdDEyMzQ=
  db_user: ZGJfZGVlcGFr
kind: Secret
...

As you can see under data section, the provided user and password are not in plain text format any more as we had with ConfigMap and instead they are base64 encoded for better security. But then these can also be decoded:

~]# echo dGVzdDEyMzQ= | base64 --decode
test1234

Declare secrets as environment variables

Now that our Secret is created, we will mount it as an environment variable in a Pod. I will create a busybox pod secret-busybox.yml with the following content in YAML file:

apiVersion: v1
kind: Pod
metadata:
  name: secret-busybox
spec:
  containers:
  - name: secret-busybox
    image: busybox
    command:
     - sleep
     - "3600"
    env:
      - name: DB_USER  ## Provide the variable name to map the value
        valueFrom:
          secretKeyRef:
            name: db-secret  ## Name of the secret created
            key: db_user  ## The key to fetch
      - name: DB_PWD  ## Provide the variable name to map the value
        valueFrom:
          secretKeyRef:
            name: db-secret  ## Name of the secret created
            key: db_password  ## The key to fetch

We have to use secretKeyRef under env to fetch the secret value from the provided key and set it as environment variable.

Let us create this Pod and make sure it is in Running state:

~]# kubectl create -f secret-busybox.yaml
pod/secret-busybox created

~]# kubectl get pods
NAME                            READY   STATUS    RESTARTS   AGE
secret-busybox                  1/1     Running   0          42s

Now we will connect to the container in this Pod and verify the environment variables which we have declared:

 ~]# kubectl exec -it secret-busybox -- printenv | grep DB
DB_USER=deepak
DB_PWD=test1234

As you can see, the variables are properly defined and the values are automatically decoded. So even if you provide the data in an encoded format, they are decoded automatically as they are passed as Kubernetes Secret.

Example-2: Declare Kubernetes Secrets manually using YAML file

In this example we will manually create Kubernetes secrets instead of using kubectl create secret command.

Manually create Kubernetes Secrets using YAML file

Following is a sample YAML file where have created two key value pairs which we will later declare as environment variable inside the Pod YAML file:

apiVersion: v1
kind: Secret
metadata:
  name: some-secret
type: Opaque
stringData:
  secret1: test123
  secret2: dummy123

apiVersion: v1
kind: Secret
metadata:
  name: some-secret-1
type: Opaque
data:
  secret1: dGVzdDEyMwo=
  secret2: ZHVtbXkxMjMK

Let us create this secret:

~]# kubectl create -f secret-1.yaml
secret/some-secret created

Now verify your data inside this secret:

 ~]# kubectl get secret some-secret -o yaml
apiVersion: v1
data:
  secret1: dGVzdDEyMw==
  secret2: ZHVtbXkxMjM=
kind: Secret
...

Declare Kubernetes Secrets as environment variables

As you can see our data is encoded and is not in plain text anymore. Now we can use this secret to declare these secrets as environment variables inside the Pod YAML.

[root@controller ~]# cat secret-busybox-1.yaml
apiVersion: v1
kind: Pod
metadata:
  name: secret-busybox-1
spec:
  containers:
  - name: secret-busybox-1
    image: busybox
    command:
     - sleep
     - "3600"
    env:
      - name: VAR1  ## Provide the variable name to map the value
        valueFrom:
          secretKeyRef:
            name: some-secret  ## Name of the secret created
            key: secret1  ## The key to fetch
      - name: VAR2  ## Provide the variable name to map the value
        valueFrom:
          secretKeyRef:
            name: some-secret  ## Name of the secret created
            key: secret2  ## The key to fetch

Let us create this Pod and make sure it is in Running state:

 ~]# kubectl create -f secret-busybox-1.yaml
pod/secret-busybox-1 created

 ~]# kubectl get pods
NAME                            READY   STATUS              RESTARTS   AGE
secret-busybox-1                1/1     Running             0          74s

Now connect to this Pod and verify if our environment variables are set properly:

~]# kubectl exec -it secret-busybox-1 -- printenv | grep VAR
VAR1=test123
VAR2=dummy123

As expected our environment variables are properly set with the values in decoded format.

Method-2: Mount Kubernetes Secrets as a file

In this section we will mount the defined Kubernetes Secrets as a file on the Pod. You can later either source these file or use them in any other possible way for your application. Here the secrets will not be defined as environment variables. This can be useful if you want to pass certificates, passphrase, key pair etc to the Pod. So you can just declare these secrets and mount them on the Pod which can be later used by the application.

Example-1: Declare Kubernetes Secrets using certificates and mount as a file

In this example we will create Kubernetes Secret to store a TLS key and certificate for application running inside the container. I have already created a self signed certificate using openssl:

[root@controller ~]# ls server.*
server.crt  server.csr  server.key

Create Kubernetes Secrets from multiple files

Create a secret named secret-tls using the create secret command and this secret data can be exposed to Pods using the secrets volume type:

[root@controller ~]# kubectl create secret generic secret-tls --from-file=server.crt --from-file=server.key
secret/secret-tls created

The secret-tls secret has been created with two data elements. Run the following command to get details:

[root@controller ~]# kubectl describe secret secret-tls
Name:         secret-tls
Namespace:    default
Labels:       <none>
Annotations:  <none>

Type:  Opaque

Data
====
server.crt:  1927 bytes
server.key:  3311 bytes

Mount Secrets as a file inside Pod's container

Next we can create our Pod where we will mount this secret volume under /tls. The file will be mounted as read-only so it cannot be modified runtime.

[root@controller ~]# cat secret-tls-pod.yml
apiVersion: v1
kind: Pod
metadata:
  name: secret-tls-pod
spec:
  containers:
    - name: secret-tls-pod
      image: busybox
      command:
      - sleep
      - "3600"
      volumeMounts:
      - name: tls-certs  ## Use this name inside volumes to define mount point
        mountPath: "/tls"  ## This will be created if not present
  volumes:
    - name: tls-certs  ## This must match the volumeMount name
      secret:
        secretName: secret-tls  ## This must match the secret name which we created earlier

In the preceding Pod specification, note that volumes are mounted the same way as we mounted the earlier ConfigMap. In the volumes section, we are instructing Kubernetes to define a volume from our Secret which we have created earlier with the certificates. In the volumeMounts section, we are defining the specific path on which Kubernetes should mount the volume

Check the status of the Pod:

[root@controller ~]# kubectl get pods secret-tls-pod
NAME             READY   STATUS    RESTARTS   AGE
secret-tls-pod   1/1     Running   0          57s

Now we can connect to this Pod and look out for our certificates under /tls:

[root@controller ~]# kubectl exec -it secret-tls-pod -- /bin/sh
/ # ls -l /tls/
total 0
lrwxrwxrwx    1 root     root            17 Jan 10 10:53 server.crt -> ..data/server.crt
lrwxrwxrwx    1 root     root            17 Jan 10 10:53 server.key -> ..data/server.key

As you can see, the container is displaying the contents of the Secret mounted onto the Pod.

Example-2: Manually declare Kubernetes Secrets and store in a file

In our previous example we took the content of the certificate files and created our secret but here we will manually create a file with our data and use it as a secret.

Create Kubernetes Secret as a file

Here we will create some-file.conf with the provided data inside the file.

~]# cat secret-mount.yaml
apiVersion: v1
kind: Secret
metadata:
  name: secret-mount
type: Opaque
stringData:
  some-file.conf: |
          VAR1=value-1
          VAR2=value-2

Let us create this secret:

 ~]# kubectl create -f secret-mount.yaml
secret/secret-mount created

Check the data content from this secret:

~]# kubectl get secret secret-mount -o yaml
apiVersion: v1
data:
  some-file.conf: VkFSMT12YWx1ZS0xClZBUjI9dmFsdWUtMgo=
kind: Secret
...

So the entire data has been encoded in base64 format and is not readable anymore.

Mount the Kubernetes Secret as a file

Now we will mount our some-file.conf under /dir1 on the Pod as read-only. Following is our YAML file required to create the Pod:

~]# cat secret-busybox-2.yaml
apiVersion: v1
kind: Pod
metadata:
  name: secret-mount
spec:
  containers:
    - name: secret-mount
      image: busybox
      command:
      - sleep
      - "3600"
      volumeMounts:
      - name: check-mount  ## Use this name inside volumes to define mount point
        mountPath: "/dir1"  ## This will be created if not present
  volumes:
    - name: check-mount  ## This must match the volumeMount name
      secret:
        secretName: secret-mount  ## This must match the secret name which we created earlier
        items:
        - key: some-file.conf  ## Existing filename used in the secret
          path: new-file-name.conf  ## New file name using which it will be mounted on the Pod

Here we have some new elements compared to what we used in previous example. We have added items section, where we have defined the key which contains the filename which we have defined in our secret file. With path we define the new filename which we want to be used on the Pod. These are optional fields and you may use it based on your requirement.

Let us create this Pod and make sure it is in Running state:

 ~]# kubectl create -f secret-busybox-2.yaml
pod/secret-mount created

~]# kubectl get pods
NAME                            READY   STATUS    RESTARTS   AGE
secret-mount                    1/1     Running   0          51s

Now that our Pod is in Running state, let us connect to the Pod and make sure our file /dir1/new-file-name.conf is created with the provided content:

[root@controller ~]# kubectl exec -it secret-mount -- sh
/ # cat /dir1/new-file-name.conf
VAR1=value-1
VAR2=value-2

So our secret is successfully mounted as a Volume with the provided file name.

Conclusion

Secret management implies managing the life cycle of secrets, which includes creating, storing, consuming, and even disposing of them safely. Secrets can be managed using a secret management software, such as Hashicorp's Vault or Square's Keywhiz. Although they can be helpful with some secret management practices, they can also bring unnecessary complexity to your system. So, they should be evaluated very carefully for the needs of your system, and full manual management should be considered as well before blindly going for a secret management tool. Regardless of the chosen method for managing secrets, some best practices should be taken into consideration. These are presented in the next section.