Simple examples to learn Kubernetes DaemonSets

Advertisement

In this tutorial we will learn about Kubernetes DaemonSets. Deployments and ReplicaSets are generally about creating a service (e.g., a web server) with multiple replicas for redundancy. But that is not the only reason you may want to replicate a set of Pods within a cluster. Another reason to replicate a set of Pods is to schedule a single Pod on every node within the cluster. Generally, the motivation for replicating a Pod to every node is to land some sort of agent or daemon on each node, and the Kubernetes object for achieving this is the DaemonSet.

 

1. Overview on Kubernetes DaemonSets

  • A Kubernetes DaemonSet ensures a copy of a Pod is running across a set of nodes in a Kubernetes cluster.
  • DaemonSets are used to deploy system daemons such as log collectors and monitoring agents, which typically must run on every node.
  • DaemonSets share similar functionality with ReplicaSets; both create Pods that are expected to be long-running services and ensure that the desired state and the observed state of the cluster match.

    Simple examples to learn Kubernetes DaemonSets
    DaemonSets run only a single pod replica on each node, whereas ReplicaSets scatter them around the whole cluster randomly.

 

2. When to use – DaemonSet and/or ReplicaSet?

Given the similarities between DaemonSets and ReplicaSets, it’s important to understand when to use one over the other.

  • ReplicaSets should be used when your application is completely decoupled from the node and you can run multiple copies on a given node without special consideration.
  • DaemonSets should be used when a single copy of your application must run on all or a subset of the nodes in the cluster.

 

3. Using a DaemonSet to run a pod on every node

A DaemonSet deploys pods to all nodes in the cluster, unless you specify that the pods should only run on a subset of all the nodes. This is done by specifying the node-Selector property in the pod template, which is part of the DaemonSet definition (similar to the pod template in a ReplicaSet or ReplicationController).

 

3.1 Create DaemonSet

In this section we will setup a logging agent using fluentd. This agent is supposed to run on all the worker nodes part of Kubernetes cluster. Before we start we would need the KIND and apiVersion value for DaemonSet.

[root@controller ~]# kubectl api-resources | grep -iE 'KIND|daemonse'
NAME                              SHORTNAMES   APIGROUP                       NAMESPACED   KIND
daemonsets                        ds           apps                           true         DaemonSet

[root@controller ~]# kubectl explain DaemonSet | head -n 2
KIND:     DaemonSet
VERSION:  apps/v1

Now that we have KIND and apiVersion, let me create the YAML file required for our DaemonSet:

[root@controller ~]# cat fluentd-daemonset.yml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluentd
  labels:
    app: fluentd
spec:
  selector:
    matchLabels:
      app: fluentd
  template:
    metadata:
      labels:
        app: fluentd
    spec:
      containers:
      - name: fluentd
        image: fluent/fluentd:v0.14.10
        resources:
          limits:
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 200Mi
        volumeMounts:
        - name: varlog
          mountPath: /var/log
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
      terminationGracePeriodSeconds: 30
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers

I have also added resource limits here as I don’t want these daemons to eat up all my system’s resources. We have already covered about how we can limit resources in a Pod

Once you have a valid DaemonSet configuration in place, you can use the kubectl create command to submit the DaemonSet to the Kubernetes API. In this section we will create a DaemonSet to ensure the fluentd HTTP server is running on every node in our cluster:

[root@controller ~]# kubectl create -f fluentd-daemonset.yml
daemonset.apps/fluentd created

 

Advertisement

3.2 Inspect DaemonSets

To list the available daemonsets we will use kubectl get command. Here ds is short abbreviation of DaemonSets.

[root@controller ~]# kubectl get ds
NAME      DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
fluentd   2         2         0       2            0                     6s

Since I have a two node cluster, we have 2 desired nodes out of which 0 are in ready state as we have just created this DS so we need to wait for few seconds before the pods are created.

Check the status of the Pods:

[root@controller ~]# kubectl get pods
NAME                            READY   STATUS              RESTARTS   AGE
fluentd-c9vcf                   0/1     ContainerCreating   0          12s
fluentd-s2lrj                   0/1     ContainerCreating   0          12s

As expected we have two new fluentd pods created where the containers are getting created. We can add -o wide to the above command to make sure both the pods are started on different worker nodes:

[root@controller ~]# kubectl get pods -o wide
NAME                            READY   STATUS    RESTARTS   AGE     IP          NODE                   NOMINATED NODE   READINESS GATES
fluentd-c9vcf                   1/1     Running   0          72s     10.44.0.3   worker-2.example.com   <none>           <none>
fluentd-s2lrj                   1/1     Running   0          72s     10.36.0.4   worker-1.example.com   <none>           <none>

This is exactly the behavior you want when managing logging daemons and other cluster-wide services. No action was required from our end; this is how the Kubernetes DaemonSet controller reconciles its observed state with our desired state.

 

4. Limiting DaemonSets to Specific Nodes

The most common use case for DaemonSets is to run a Pod across every node in a Kubernetes cluster. However, there are some cases where you want to deploy a Pod to only a subset of nodes. For example, maybe you have a workload that requires a GPU or access to fast storage only available on a subset of nodes in your cluster. In cases like these, node labels can be used to tag specific nodes that meet workload requirements.

 

4.1 Create DaemonSet using Node Selectors

Node selectors can be used to limit what nodes a Pod can run on in a given Kubernetes cluster. Node selectors are defined as part of the Pod spec when creating a DaemonSet. The DaemonSet configuration in following YAMl file limits NGINX to running only on nodes with the ssd=true label set.

[root@controller ~]# cat nginx-daemonset.yml
apiVersion: apps/v1
kind: "DaemonSet"
metadata:
  labels:
    app: nginx
    ssd: "true"
  name: nginx-fast-storage
spec:
  selector:
    matchLabels:
      ssd: "true"
  template:
    metadata:
      labels:
        app: nginx
        ssd: "true"
    spec:
      nodeSelector:
        ssd: "true"
      containers:
        - name: nginx
          image: nginx:1.10.0

We are defining a DaemonSet that will run a pod with a single container based on the nginx container image. An instance of this pod will be created for each node that has the ssd: true label.

Let’s create our nginx daemon set:

Advertisement
[root@controller ~]# kubectl create -f nginx-daemonset.yml
daemonset.apps/nginx-fast-storage created

 

4.2 Inspect DaemonSet

List the available daemon sets:

[root@controller ~]# kubectl get ds
NAME                 DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
fluentd              2         2         2       2            2                     5m37s
nginx-fast-storage   0         0         0       0            0           ssd=true        5s

So the nginx-fast-storage DaemonSet has 0 DESIRED nodes, this would mean that currently there are no active cluster nodes with ssd: true label. You can list the nodes and their labels using:

[root@controller ~]# kubectl get nodes --show-labels
NAME                     STATUS   ROLES    AGE   VERSION   LABELS
controller.example.com   Ready    master   42d   v1.19.3   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=controller.example.com,kubernetes.io/os=linux,node-role.kubernetes.io/master=
worker-1.example.com     Ready    <none>   42d   v1.19.3   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=worker-1.example.com,kubernetes.io/os=linux
worker-2.example.com     Ready    <none>   42d   v1.19.3   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=worker-2.example.com,kubernetes.io/os=linux

 

4.3 Adding required labels to the node

Let us go ahead and add this label to one of our worker nodes:

[root@controller ~]# kubectl label nodes worker-1.example.com ssd=true
node/worker-1.example.com labeled

Now that one of our worker node is labelled with ssd: true, let’s re-validate the status of DaemonSets:

[root@controller ~]# kubectl get ds
NAME                 DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
fluentd              2         2         2       2            2                     8m34s
nginx-fast-storage   1         1         0       1            0           ssd=true        3m2s

As expected, now we have one DESIRED node where the label has matched, this would mean that now one Pod would be created on worker-1 node:

[root@controller ~]# kubectl get pods -o wide
NAME                            READY   STATUS              RESTARTS   AGE     IP          NODE                   NOMINATED NODE   READINESS GATES
fluentd-c9vcf                   1/1     Running             0          8m39s   10.44.0.3   worker-2.example.com   <none>           <none>
fluentd-s2lrj                   1/1     Running             0          8m39s   10.36.0.4   worker-1.example.com   <none>           <none>
nginx-fast-storage-4wmgc        0/1     ContainerCreating   0          8s      <none>      worker-1.example.com   <none>           <none>

 

4.5 Removing the required label from the node

Now, imagine you’ve made a mistake and have mislabeled one of the nodes. It has a spinning disk drive, not an SSD. What happens if you change the node’s label?

To remove a label from the node we use following syntax:

kubectl label node <nodename> <labelname>-

So to remove the ssd label from worker-1.example.com, we will use:

[root@controller ~]# kubectl label node worker-1.example.com ssd-
node/worker-1.example.com labeled

As soon as we remove the label, we can see that the Pod status is Terminating:

Advertisement
[root@controller ~]# kubectl get pods -o wide
NAME                            READY   STATUS        RESTARTS   AGE     IP          NODE                   NOMINATED NODE   READINESS GATES
fluentd-c9vcf                   1/1     Running       0          73m     10.44.0.3   worker-2.example.com   <none>           <none>
fluentd-s2lrj                   1/1     Running       0          73m     10.36.0.4   worker-1.example.com   <none>           <none>
nginx-fast-storage-4wmgc        0/1     Terminating   0          65m     <none>      worker-1.example.com   <none>           <none>

So this is one good thing with DaemonSets i.e. you don’t have to worry about manually deleting the pods in such scenarios. This wraps up our exploration of DaemonSets, so you may want to delete the DaemonSets which we created in this article. If you still have any other daemon pods running, you’ll see that deleting the DaemonSet deletes those pods as well.

 

5. Deleting DaemonSets

Deleting a DaemonSet is pretty straightforward using the kubectl delete command. Just be sure to supply the correct name of the DaemonSet you would like to delete. If you don’t know the name of the available DaemonSets then you can always list them:

[root@controller ~]# kubectl get ds
NAME                 DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
fluentd              2         2         2       2            2           <none>          80m
nginx-fast-storage   0         0         0       0            0           ssd=true        75m

Now that we have the names of available DaemonSets, lets delete them:

[root@controller ~]# kubectl delete ds fluentd nginx-fast-storage
daemonset.apps "fluentd" deleted
daemonset.apps "nginx-fast-storage" deleted
WARNING:

Deleting a DaemonSet will also delete all the Pods being managed by that DaemonSet. Set the --cascade flag to false to ensure only the DaemonSet is deleted and not the Pods.

 

6. Conclusion

DaemonSets provide an easy-to-use abstraction for running a set of Pods on every node in a Kubernetes cluster, or, if the case requires it, on a subset of nodes based on labels. The DaemonSet provides its own controller and scheduler to ensure key services like monitoring agents are always up and running on the right nodes in your cluster.

For some applications, you simply want to schedule a certain number of replicas; you don’t really care where they run as long as they have sufficient resources and distribution to operate reliably. However, there is a different class of applications, like agents and monitoring applications, that need to be present on every machine in a cluster to function properly. These DaemonSets aren’t really traditional serving applications, but rather add additional capabilities and features to the Kubernetes cluster itself.

If my articles on GoLinuxCloud has helped you, kindly consider buying me a coffee as a token of appreciation.

Buy GoLinuxCloud a Coffee

For any other feedbacks or questions you can either use the comments section or contact me form.

Thank You for your support!!

Leave a Comment