SOLVED: Update ConfigMap & Secrets without Pod restart in K8s

Introduction

Most applications are designed to loosely couple code, configuration, and data. The configuration files and data are not hard-coded as part of the application code. Instead, the configuration and data are loaded from an external source. This allows for deploying the application to different environments without requiring any change in the source code. Kubernetes uses the concept of secrets and configmaps to decouple configuration information from container images.

Applications often require access to sensitive information. For instance, a backend web application will require access to database credentials to perform a database query. To store sensitive information like this, secrets are used. Configmaps are used in cases when data is not of sensitive nature. The information contained in a configmap does not require protection and data is stored in plain text. Data in a secret is not stored in plain text, rather it is Base64 encoded.

 

Known Limitation

At the time of writing this article, Kubernetes has following known limitations to support runtime update of configmap and secrets without restart:

• Configmap and/or secrets when they are mounted as volumes. If these resources are used as environment variables then runtime update is not supported.
• A container using a ConfigMap as a subPath volume mount will not receive ConfigMap updates.

 

How runtime update works for ConfigMaps and Secrets without restart?

When a secret or configmap is mounted as volume then a symbolic link is created in the mount path. So every time a secret or configmap is modified, the symbolic link will start pointing to the new data while the old one is deleted.

From official documentation

When a ConfigMap currently consumed in a volume is updated, projected keys are eventually updated as well. The kubelet checks whether the mounted ConfigMap is fresh on every periodic sync. However, the kubelet uses its local cache for getting the current value of the ConfigMap. The type of the cache is configurable using the ConfigMapAndSecretChangeDetectionStrategy field in the KubeletConfiguration struct. A ConfigMap can be either propagated by watch (default), ttl-based, or by redirecting all requests directly to the API server. As a result, the total delay from the moment when the ConfigMap is updated to the moment when new keys are projected to the Pod can be as long as the kubelet sync period + cache propagation delay, where the cache propagation delay depends on the chosen cache type (it equals to watch propagation delay, ttl of cache, or zero correspondingly).

 

Prerequisites

• You must have a working Kubernetes Cluster.
• Working knowledge of K8s and yaml language

 

Setup Lab environment

The lab environment consists of one master node and two worker nodes, running K8s version 1.23. However, there aren’t any strict requirements and the steps should work on any K8s setup.

 

Create Secret

We will create a dummy secret which we will use to test the runtime update on our deployment:

I already have a SSH key which I will copy to my current PATH and then create a secret using CLI command:

]# cp ~/.ssh/id_rsa ssh-privatekey

]# kubectl create secret generic secret1 --from-file=ssh-privatekey --type=kubernetes.io/ssh-auth
secret/secret1 created

Verify the secret is created:

 

Create ConfigMap

Let's also create one ConfigMap to demonstrate the runtime updates:

]# cat file.txt 
samplevar: samplevalue

]# kubectl create configmap cm1 --from-file=file.txt 
configmap/cm1 created

]# kubectl get cm
NAME               DATA   AGE
cm1                1      3s

So now we have a new configmap "cm1" created in our default namespace.

 

Create Deployment

We will need a deployment or set of pods on which we can validate the steps from this article. I will create a small deployment using nginx and 1 replicaset and mount our configmap and secrets as volumeMounts.

apiVersion: apps/v1
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: nginx
  name: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - image: bcmt-registry:5000/secure-ssh:sudo
        name: nginx
        volumeMounts:
        - name: cm-volume
          mountPath: /etc/config
        - name: secret-volume
          mountPath: /etc/sshconfig
      volumes:
      - name: cm-volume
        configMap:
          name: cm1
      - name: secret-volume
        secret:
          secretName: secret1

Create the deployment:

kubectl create -f nginx.yml

Verify if nginx pod is up and running.

]# kubectl get pods 
NAME                     READY   STATUS    RESTARTS   AGE
nginx-578dcf9879-bpp5m   1/1     Running   0          64s

 

Update ConfigMap automatically without pod restart

Next let's verify if a pod is able to get the updated COnfigMap content wihtout going for restart. I will update the configmap content using kubectl edit:

Save and exit the file. The kubectl edit will open the configmap for write purpose using the default EDITOR on your distribution.

Now verify the content of your configmap. We had mounted it on /etc/config/file.txt

[root@fi-758-ncs20fp2-8-cluster-01-cs-01 tmp]# kubectl exec -it nginx-686b7b4785-p92qz -- cat /etc/config/file.txt
samplevar: value1

As expected, the new value for the variable has been picked without any pod restart:

[root@fi-758-ncs20fp2-8-cluster-01-cs-01 tmp]# kubectl get pods
NAME                     READY   STATUS    RESTARTS   AGE
nginx-686b7b4785-p92qz   1/1     Running   0          5m1s

 

Update Secrets automatically without any Pod restart

Similarly let us also try to update the Kubernetes Secret which we created in the previous steps using kubectl edit command. For the sake ofexplanation, I have created a new SSH key:

# ssh-keygen -t rsa -f new-ssh-privatekey -P ""

Then converted the private key into base64 format

# cat new-ssh-privatekey | base64  --wrap=0
LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFcEFJQkFBS0NBUUVBeENncFVHaXFPN2orN2wweWV5VEUyTHZRSWRqOW1TVndRc2hGVFVGUVE1TEVLeUlaClVsVWVHWWVXMWFycThHazA1UGNNaGRGWWUyQ1BSdTR3Yzgvc3grakJLRXdUcFVneGkvemZYcVhTaWExRkVucHkKTzBWQWlPOG9NQWg2RC9BbEswWDVDd0NxeG1SaC9KaVcxUTM3ZEN4djRmZnRZRjlUVjBTL3JPeUhqOTdUaUdFRwpMa2hIREpMaGVIUFVoVndNdzIyQnpDTEdtVE1YZVJTTk02TnNCNmhIOFNBamN1TTk0ZDhJZzJLRXh4aVVNY3FkCjd5SDhMYTFXVHZveTRzd3FnV0w5WnNScjNDZzY5TDRxTk56b0JoUEtybzI1a1RweE9iZG1PMzV3WUpTODFTNFYKNkRsdEFlV1FzNzZGR285LzI0b2Q2RVdpak14d281TjFKZVVsd1FJREFRQUJBb0lCQUN5WFBKME16Zlg1bmVvdAp3WFlBNjhhaEd6VTJrSitweFJWSlZZZTBXenloTm5yZnE0WHQxNFBTTU5XdG51NjcyOHhZNUwzZTB4Qm82T2trCjZGckxYM1lxVVE2S0RNVTczaGVHaW5pSGxZNjZsc01XbHJVbWp2OFI3cjdNam9MbEFtNE40QWxDUTVBSjdjUncKSTRtWFBod3dwZFptZDgyNm5jVnUyV3ZEOFNVaENtczQ0aldmcFVteFFNSjZuaUo4RkthdUp1MVc3ZEJubnhXUQpRY2Q2TUJuYU0xalVPUlRBWDdFVmR0Y0JwaFVDTU1vTnZSMEpmR040cGJ5elViSFN4MXFudlJOblNRQk1Pa2xlCjZueWNCL3p0M3hYVFpsekVtYVdoVVA0ajhSbDhqdkdiNVlCbVA0aDJPMlBhVUo3ZmJMWFk3VFk5ZEl1eE05ajcKd21NVldXRUNnWUVBNUE3YkV4RVdwcHBmMXAyUVNFeTFzRjBrNTQ4NkFvamFiLzF6MlIxYU40My9QaERTaWpKZgpXL1pJV2Q3NkplcjRSd2g1YmZsUTR1S3U1bzk4UjBBYVQ3TytValVjeC9jNmhjV2RxNUhTQS92SCtWamVHVjJhCmpWVkdTeXVmTzNTRnRJbnZZYUlpVUJ0WVVxYjQ1VFYyQWhkZnJucHRKbTFCeDA3VVIrWXJYMDBDZ1lFQTNEQzIKSkZwd09sR1RabGc4Y2RmRDl6NktvTmhha2RTcEZQUzIwQkQrZHNTL1FPRk9vWlloLzJIVFlDeVlpTStMRElOSAo3Q3pUL3RrbkNaN1RObTdUcHREU0VudE1GRm12eExERjU5ZXZGZy9tZ09MK3MzcUtwTzJSSUFVcTdpdVRLbzRUCkNmcm1SMHdjUHdLOTZ4SFBiLzlpNnAxbDdSRmpBb0tpTG9wWFRrVUNnWUJNckxuM0ZSMjZjZGlhL1dxUEJFdHAKdWtjNEd5MXp3TE5BUjhSMVVLc09WbzFrUHArcW12ajRvRHIvRERxcUdPL1VZZ01CZUhzN2JOOUU0U1QxaDVYUgpDaXVJMUJhVEhJbnVnOXhZM0xQeFp1dDY1K2YwTzBaRkVsQ0o0V2F0eEtWWFo3QzE4Sjc4czlUa0pRTTFmTjNxCkloV25RYjRFMTJMd01ZNnBoYmM3V1FLQmdRQ0szUUdScmFPSGMvamttNU1MTE1yK3UyZU1Cc1lmb0NFK0FSTGwKNTBIRHYxTHFaTzFGQkx6T0pYQzcvNFAzREFTaVVJemtTbVVzSE9EOHRUaDQ1SzRBVDBPY3VqdUJ2Z29Xbm5GQgpSSW03L1MwZWJZbTV3UGQ5Q2dIelVxNy9ZMlc5ZWJwU0dmUnVWSGFmMm1mUnZ2cTJwRFpLeGhjSXltVkpxUDhGCklPUHNqUUtCZ1FESnQ3bGxyNGIrV1pBUWNFbjdCUEdKaHZVUjF2SjN1V1QwZ3d0TUFvYXdXWk5QNm8vL3B5cXgKaVdRNjhBbFMvT0tuVXBKWEc1TTdCajEvM0N5azNRenU0RnNMbnZhVUp1UzdFUVN1L0F4dnhtZU9CR2p6Yy92YgoxTCt1b3Y3WjkvR0dSU1VaRy9mb1M1QkdDNW9hTDVRNXlZZkNLNzl1ZmFrNi9jZzZKS1FKbWc9PQotLS0tLUVORCBSU0EgUFJJVkFURSBLRVktLS0tLQo=

Now using kubectl edit I will edit the secret1 and add the updated value for my private key:

]# kubectl edit secret/secret1
# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: v1
data:
  ssh-privatekey: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFcEFJQkFBS0NBUUVBeENncFVHaXFPN2orN2wweWV5VEUyTHZRSWRqOW1TVndRc2hGVFVGUVE1TEVLeUlaClVsVWVHWWVXMWFycThHazA1UGNNaGRGWWUyQ1BSdTR3Yzgvc3grakJLRXdUcFVneGkvemZYcVhTaWExRkVucHkKTzBWQWlPOG9NQWg2RC9BbEswWDVDd0NxeG1SaC9KaVcxUTM3ZEN4djRmZnRZRjlUVjBTL3JPeUhqOTdUaUdFRwpMa2hIREpMaGVIUFVoVndNdzIyQnpDTEdtVE1YZVJTTk02TnNCNmhIOFNBamN1TTk0ZDhJZzJLRXh4aVVNY3FkCjd5SDhMYTFXVHZveTRzd3FnV0w5WnNScjNDZzY5TDRxTk56b0JoUEtybzI1a1RweE9iZG1PMzV3WUpTODFTNFYKNkRsdEFlV1FzNzZGR285LzI0b2Q2RVdpak14d281TjFKZVVsd1FJREFRQUJBb0lCQUN5WFBKME16Zlg1bmVvdAp3WFlBNjhhaEd6VTJrSitweFJWSlZZZTBXenloTm5yZnE0WHQxNFBTTU5XdG51NjcyOHhZNUwzZTB4Qm82T2trCjZGckxYM1lxVVE2S0RNVTczaGVHaW5pSGxZNjZsc01XbHJVbWp2OFI3cjdNam9MbEFtNE40QWxDUTVBSjdjUncKSTRtWFBod3dwZFptZDgyNm5jVnUyV3ZEOFNVaENtczQ0aldmcFVteFFNSjZuaUo4RkthdUp1MVc3ZEJubnhXUQpRY2Q2TUJuYU0xalVPUlRBWDdFVmR0Y0JwaFVDTU1vTnZSMEpmR040cGJ5elViSFN4MXFudlJOblNRQk1Pa2xlCjZueWNCL3p0M3hYVFpsekVtYVdoVVA0ajhSbDhqdkdiNVlCbVA0aDJPMlBhVUo3ZmJMWFk3VFk5ZEl1eE05ajcKd21NVldXRUNnWUVBNUE3YkV4RVdwcHBmMXAyUVNFeTFzRjBrNTQ4NkFvamFiLzF6MlIxYU40My9QaERTaWpKZgpXL1pJV2Q3NkplcjRSd2g1YmZsUTR1S3U1bzk4UjBBYVQ3TytValVjeC9jNmhjV2RxNUhTQS92SCtWamVHVjJhCmpWVkdTeXVmTzNTRnRJbnZZYUlpVUJ0WVVxYjQ1VFYyQWhkZnJucHRKbTFCeDA3VVIrWXJYMDBDZ1lFQTNEQzIKSkZwd09sR1RabGc4Y2RmRDl6NktvTmhha2RTcEZQUzIwQkQrZHNTL1FPRk9vWlloLzJIVFlDeVlpTStMRElOSAo3Q3pUL3RrbkNaN1RObTdUcHREU0VudE1GRm12eExERjU5ZXZGZy9tZ09MK3MzcUtwTzJSSUFVcTdpdVRLbzRUCkNmcm1SMHdjUHdLOTZ4SFBiLzlpNnAxbDdSRmpBb0tpTG9wWFRrVUNnWUJNckxuM0ZSMjZjZGlhL1dxUEJFdHAKdWtjNEd5MXp3TE5BUjhSMVVLc09WbzFrUHArcW12ajRvRHIvRERxcUdPL1VZZ01CZUhzN2JOOUU0U1QxaDVYUgpDaXVJMUJhVEhJbnVnOXhZM0xQeFp1dDY1K2YwTzBaRkVsQ0o0V2F0eEtWWFo3QzE4Sjc4czlUa0pRTTFmTjNxCkloV25RYjRFMTJMd01ZNnBoYmM3V1FLQmdRQ0szUUdScmFPSGMvamttNU1MTE1yK3UyZU1Cc1lmb0NFK0FSTGwKNTBIRHYxTHFaTzFGQkx6T0pYQzcvNFAzREFTaVVJemtTbVVzSE9EOHRUaDQ1SzRBVDBPY3VqdUJ2Z29Xbm5GQgpSSW03L1MwZWJZbTV3UGQ5Q2dIelVxNy9ZMlc5ZWJwU0dmUnVWSGFmMm1mUnZ2cTJwRFpLeGhjSXltVkpxUDhGCklPUHNqUUtCZ1FESnQ3bGxyNGIrV1pBUWNFbjdCUEdKaHZVUjF2SjN1V1QwZ3d0TUFvYXdXWk5QNm8vL3B5cXgKaVdRNjhBbFMvT0tuVXBKWEc1TTdCajEvM0N5azNRenU0RnNMbnZhVUp1UzdFUVN1L0F4dnhtZU9CR2p6Yy92YgoxTCt1b3Y3WjkvR0dSU1VaRy9mb1M1QkdDNW9hTDVRNXlZZkNLNzl1ZmFrNi9jZzZKS1FKbWc9PQotLS0tLUVORCBSU0EgUFJJVkFURSBLRVktLS0tLQo=
kind: Secret
metadata:
  creationTimestamp: "2022-06-01T06:19:37Z"
  name: secret1
  namespace: default
  resourceVersion: "19944036"
  selfLink: /api/v1/namespaces/default/secrets/secret1
  uid: fcc91312-72d7-479a-9353-419e1c8dff64
type: kubernetes.io/ssh-auth

Save and exit the file.

Now verify the content of your secret file inside the Pod:

As expected, the new value for the variable has been picked without any pod restart:

[root@fi-758-ncs20fp2-8-cluster-01-cs-01 tmp]# kubectl get pods
NAME                     READY   STATUS    RESTARTS   AGE
nginx-686b7b4785-p92qz   1/1     Running   0          15m1s

 

Update ConfigMap/Secret as environment variable with Pod restart

Now if you have defined any ConfigMap or Secret as Environment Variable then updating them would restart the pod automatically. Here is an example:

I will create one more configmap:

# kubectl create configmap cm2 --from-literal=color=blue
configmap/cm2 created

Now I will attach this configmap to our existing nginx deployment as environment variable:

]# kubectl set env deployment nginx --from=configmap/cm1
deployment.apps/nginx env updated

 

Now as soon as I hit this command, I can see my nginx pods are going for restart to add the new environment variable:

 

Summary

Mounting the configmap and secrets as volumes inside the container gives the end user a lot of flexibility as these values can be changed and updated inside the application on the fly. However, there’s a catch when using this approach. When we specify a mount path which doesn’t exist inside the container, such as /etc/config or /etc/sshconfig, it is created automatically inside the container. However, if the path already exists inside the container, all the current files and directories inside that path will be overwritten. To avoid this, the subPath option is used inside the volumeMounts section.