In this tutorial we will learn about Kubernetes DaemonSets. Deployments and ReplicaSets are generally about creating a service (e.g., a web server) with multiple replicas for redundancy. But that is not the only reason you may want to replicate a set of Pods within a cluster. Another reason to replicate a set of Pods is to schedule a single Pod on every node within the cluster. Generally, the motivation for replicating a Pod to every node is to land some sort of agent or daemon on each node, and the Kubernetes object for achieving this is the DaemonSet.
1. Overview on Kubernetes DaemonSets
- A Kubernetes DaemonSet ensures a copy of a Pod is running across a set of nodes in a Kubernetes cluster.
- DaemonSets are used to deploy system daemons such as log collectors and monitoring agents, which typically must run on every node.
- DaemonSets share similar functionality with ReplicaSets; both create Pods that are expected to be long-running services and ensure that the desired state and the observed state of the cluster match.
DaemonSets run only a single pod replica on each node, whereas ReplicaSets scatter them around the whole cluster randomly.
2. When to use - DaemonSet and/or ReplicaSet?
Given the similarities between DaemonSets and ReplicaSets, it’s important to understand when to use one over the other.
- ReplicaSets should be used when your application is completely decoupled from the node and you can run multiple copies on a given node without special consideration.
- DaemonSets should be used when a single copy of your application must run on all or a subset of the nodes in the cluster.
3. Using a DaemonSet to run a pod on every node
A DaemonSet deploys pods to all nodes in the cluster, unless you specify that the pods should only run on a subset of all the nodes. This is done by specifying the node-Selector property in the pod template, which is part of the DaemonSet definition (similar to the pod template in a ReplicaSet or ReplicationController).
3.1 Create DaemonSet
In this section we will setup a logging agent using fluentd. This agent is supposed to run on all the worker nodes part of Kubernetes cluster. Before we start we would need the KIND and apiVersion value for DaemonSet.
[root@controller ~]# kubectl api-resources | grep -iE 'KIND|daemonse' NAME SHORTNAMES APIGROUP NAMESPACED KIND daemonsets ds apps true DaemonSet [root@controller ~]# kubectl explain DaemonSet | head -n 2 KIND: DaemonSet VERSION: apps/v1
Now that we have KIND and apiVersion, let me create the YAML file required for our DaemonSet:
[root@controller ~]# cat fluentd-daemonset.yml apiVersion: apps/v1 kind: DaemonSet metadata: name: fluentd labels: app: fluentd spec: selector: matchLabels: app: fluentd template: metadata: labels: app: fluentd spec: containers: - name: fluentd image: fluent/fluentd:v0.14.10 resources: limits: memory: 200Mi requests: cpu: 100m memory: 200Mi volumeMounts: - name: varlog mountPath: /var/log - name: varlibdockercontainers mountPath: /var/lib/docker/containers readOnly: true terminationGracePeriodSeconds: 30 volumes: - name: varlog hostPath: path: /var/log - name: varlibdockercontainers hostPath: path: /var/lib/docker/containers
I have also added resource limits here as I don't want these daemons to eat up all my system's resources. We have already covered about how we can limit resources in a Pod
Once you have a valid DaemonSet configuration in place, you can use the kubectl create command to submit the DaemonSet to the Kubernetes API. In this section we will create a DaemonSet to ensure the fluentd HTTP server is running on every node in our cluster:
[root@controller ~]# kubectl create -f fluentd-daemonset.yml
daemonset.apps/fluentd created
3.2 Inspect DaemonSets
To list the available daemonsets we will use kubectl get command. Here ds is short abbreviation of DaemonSets.
[root@controller ~]# kubectl get ds
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
fluentd 2 2 0 2 0 6s
Since I have a two node cluster, we have 2 desired nodes out of which 0 are in ready state as we have just created this DS so we need to wait for few seconds before the pods are created.
Check the status of the Pods:
[root@controller ~]# kubectl get pods NAME READY STATUS RESTARTS AGE fluentd-c9vcf 0/1 ContainerCreating 0 12s fluentd-s2lrj 0/1 ContainerCreating 0 12s
As expected we have two new fluentd pods created where the containers are getting created. We can add -o wide to the above command to make sure both the pods are started on different worker nodes:
[root@controller ~]# kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES fluentd-c9vcf 1/1 Running 0 72s 10.44.0.3 worker-2.example.com <none> <none> fluentd-s2lrj 1/1 Running 0 72s 10.36.0.4 worker-1.example.com <none> <none>
This is exactly the behavior you want when managing logging daemons and other cluster-wide services. No action was required from our end; this is how the Kubernetes DaemonSet controller reconciles its observed state with our desired state.
4. Limiting DaemonSets to Specific Nodes
The most common use case for DaemonSets is to run a Pod across every node in a Kubernetes cluster. However, there are some cases where you want to deploy a Pod to only a subset of nodes. For example, maybe you have a workload that requires a GPU or access to fast storage only available on a subset of nodes in your cluster. In cases like these, node labels can be used to tag specific nodes that meet workload requirements.
4.1 Create DaemonSet using Node Selectors
Node selectors can be used to limit what nodes a Pod can run on in a given Kubernetes cluster. Node selectors are defined as part of the Pod spec when creating a DaemonSet. The DaemonSet configuration in following YAMl file limits NGINX to running only on nodes with the ssd=true label set.
[root@controller ~]# cat nginx-daemonset.yml apiVersion: apps/v1 kind: "DaemonSet" metadata: labels: app: nginx ssd: "true" name: nginx-fast-storage spec: selector: matchLabels: ssd: "true" template: metadata: labels: app: nginx ssd: "true" spec: nodeSelector: ssd: "true" containers: - name: nginx image: nginx:1.10.0
We are defining a DaemonSet that will run a pod with a single container based on the nginx container image. An instance of this pod will be created for each node that has the ssd: true label.
Let's create our nginx daemon set:
[root@controller ~]# kubectl create -f nginx-daemonset.yml
daemonset.apps/nginx-fast-storage created
4.2 Inspect DaemonSet
List the available daemon sets:
[root@controller ~]# kubectl get ds
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
fluentd 2 2 2 2 2 5m37s
nginx-fast-storage 0 0 0 0 0 ssd=true 5s
So the nginx-fast-storage DaemonSet has 0 DESIRED nodes, this would mean that currently there are no active cluster nodes with ssd: true label. You can list the nodes and their labels using:
[root@controller ~]# kubectl get nodes --show-labels
NAME STATUS ROLES AGE VERSION LABELS
controller.example.com Ready master 42d v1.19.3 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=controller.example.com,kubernetes.io/os=linux,node-role.kubernetes.io/master=
worker-1.example.com Ready <none> 42d v1.19.3 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=worker-1.example.com,kubernetes.io/os=linux
worker-2.example.com Ready <none> 42d v1.19.3 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=worker-2.example.com,kubernetes.io/os=linux
4.3 Adding required labels to the node
Let us go ahead and add this label to one of our worker nodes:
[root@controller ~]# kubectl label nodes worker-1.example.com ssd=true
node/worker-1.example.com labeled
Now that one of our worker node is labelled with ssd: true, let's re-validate the status of DaemonSets:
[root@controller ~]# kubectl get ds
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
fluentd 2 2 2 2 2 8m34s
nginx-fast-storage 1 1 0 1 0 ssd=true 3m2s
As expected, now we have one DESIRED node where the label has matched, this would mean that now one Pod would be created on worker-1 node:
[root@controller ~]# kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES fluentd-c9vcf 1/1 Running 0 8m39s 10.44.0.3 worker-2.example.com <none> <none> fluentd-s2lrj 1/1 Running 0 8m39s 10.36.0.4 worker-1.example.com <none> <none> nginx-fast-storage-4wmgc 0/1 ContainerCreating 0 8s <none> worker-1.example.com <none> <none>
4.5 Removing the required label from the node
Now, imagine you’ve made a mistake and have mislabeled one of the nodes. It has a spinning disk drive, not an SSD. What happens if you change the node’s label?
To remove a label from the node we use following syntax:
kubectl label node <nodename> <labelname>-
So to remove the ssd label from worker-1.example.com, we will use:
[root@controller ~]# kubectl label node worker-1.example.com ssd-
node/worker-1.example.com labeled
As soon as we remove the label, we can see that the Pod status is Terminating:
[root@controller ~]# kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
fluentd-c9vcf 1/1 Running 0 73m 10.44.0.3 worker-2.example.com <none> <none>
fluentd-s2lrj 1/1 Running 0 73m 10.36.0.4 worker-1.example.com <none> <none>
nginx-fast-storage-4wmgc 0/1 Terminating 0 65m <none> worker-1.example.com <none> <none>
So this is one good thing with DaemonSets i.e. you don't have to worry about manually deleting the pods in such scenarios. This wraps up our exploration of DaemonSets, so you may want to delete the DaemonSets which we created in this article. If you still have any other daemon pods running, you’ll see that deleting the DaemonSet deletes those pods as well.
5. Deleting DaemonSets
Deleting a DaemonSet is pretty straightforward using the kubectl delete command. Just be sure to supply the correct name of the DaemonSet you would like to delete. If you don't know the name of the available DaemonSets then you can always list them:
[root@controller ~]# kubectl get ds NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE fluentd 2 2 2 2 2 <none> 80m nginx-fast-storage 0 0 0 0 0 ssd=true 75m
Now that we have the names of available DaemonSets, lets delete them:
[root@controller ~]# kubectl delete ds fluentd nginx-fast-storage
daemonset.apps "fluentd" deleted
daemonset.apps "nginx-fast-storage" deleted
DaemonSet will also delete all the Pods being managed by that DaemonSet. Set the --cascade flag to false to ensure only the DaemonSet is deleted and not the Pods.
6. Conclusion
DaemonSets provide an easy-to-use abstraction for running a set of Pods on every node in a Kubernetes cluster, or, if the case requires it, on a subset of nodes based on labels. The DaemonSet provides its own controller and scheduler to ensure key services like monitoring agents are always up and running on the right nodes in your cluster.
For some applications, you simply want to schedule a certain number of replicas; you don’t really care where they run as long as they have sufficient resources and distribution to operate reliably. However, there is a different class of applications, like agents and monitoring applications, that need to be present on every machine in a cluster to function properly. These DaemonSets aren’t really traditional serving applications, but rather add additional capabilities and features to the Kubernetes cluster itself.
