Helm hook-weight examples to order Jobs in Kubernetes

We already discussed in depth about usage of hooks in helm charts with different examples. Although we did overlook one of the parameters i.e. hook-weight in all our examples.

 

Overview on hook-weight in helm charts

• Hooks can be weighted and specify a deletion policy for the resources after they have run.
• The weight enables more than one hook for the same event to be specified while providing an order in which they will run.
• This gives you the ability to ensure a deterministic order.
• Because Kubernetes resources are used for the execution of hooks, the resources are stored in Kubernetes even after execution has completed.
• The weight, specified by the helm.sh/hook-weight annotation key, is a number represented as a string.
• It should always be a string.
• The weight can be a positive or negative number and has a default value of 0.
• For example, a resource with the annotation "helm.sh/hook-weight": "-5" would be run before "helm.sh/hook-weight": "5", but would be run after a resource with the annotation "helm.sh/hook-weight": "-10".
• Prior to executing hooks, Helm sorts them in ascending order.

 

Example-1: Define job order using helm hook-weight

In this example we will create a helm chart with multiple pre-install hook and define a certain order in which each of those hooks must execute.

 

Create a chart

First let me create a chart, I create all my charts under /helm-charts directory:

[root@controller ~]# cd /helm-charts/

[root@controller helm-charts]# helm create chart-5
Creating chart-5

As we don't need the existing template files, I will just delete them so that we can create our own helm chart templates:

[root@controller helm-charts]# rm -rf chart-5/templates/*

[root@controller helm-charts]# tree chart-5/
chart-5/
├── charts
├── Chart.yaml
├── templates
└── values.yaml

2 directories, 2 files

 

Create helm hook templates

Here is my first pre-install job with "hook-weight": "-2". We will create a busybox container which will echo "pre-install-job-hook-1 running, weight -2" and sleep for 2 seconds. The backoffLimit to specify the number of retries before considering a Job as failed. The back-off limit is set by default to 6.

[root@controller helm-charts]# cat chart-5/templates/pre-install-job-hook-1.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: pre-install-hook-1
  annotations:
    "helm.sh/hook": "pre-install"
    "helm.sh/hook-weight": "-2"
spec:
  template:
    spec:
      containers:
       - name: pre-install
         image: busybox
         imagePullPolicy: IfNotPresent
         command: ['sh', '-c', 'echo pre-install-job-hook-1 running, weight -2 ; sleep 2']
      restartPolicy: OnFailure
      terminationGracePeriodSeconds: 0
  backoffLimit: 3
  completions: 1
  parallelism: 1

Similarly we will create second pre-install job with "hook-weight": "3"

 

And our last pre-install job will have "hook-weight": "5"

 

In a helm chart we would also need a main pod. So I will create one statefulset pod with single replica at chart-5/templates/statefulset.yaml.

 

This is the final tree structure with all the files for my chart-5:

[root@controller helm-charts]# tree chart-5/
chart-5/
├── charts
├── Chart.yaml
├── templates
│   ├── pre-install-job-hook-1.yaml
│   ├── pre-install-job-hook-2.yaml
│   ├── pre-install-job-hook-3.yaml
│   └── statefulset.yaml
└── values.yaml

2 directories, 6 files

 

Perform pre-checks on the helm chart

Next we will lint the chart to make sure our template files have no incorrect values:

[root@controller helm-charts]# helm lint chart-5/
==> Linting chart-5/
[INFO] Chart.yaml: icon is recommended

1 chart(s) linted, 0 chart(s) failed

As the lint was success, next we will attempt to install the chart using --dry-run to find out any issues before actually installing it (output trimmed in the output below):

[root@controller helm-charts]# helm install hook-weight --dry-run chart-5/
NAME: hook-weight
LAST DEPLOYED: Thu Mar 18 23:24:09 2021
NAMESPACE: default
STATUS: pending-install
REVISION: 1
TEST SUITE: None
HOOKS:
---
# Source: chart-5/templates/pre-install-job-hook-1.yaml
apiVersion: batch/v1
kind: Job
...

 

Install helm chart

The dry run execution was also successful so it is safe to install the helm chart:

[root@controller helm-charts]# helm install hook-weight chart-5/
NAME: hook-weight
LAST DEPLOYED: Thu Mar 18 23:24:13 2021
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None

The help chart has been successfully deployed.

 

Verify helm hook-weight order

Next check the list of job and their execution status:

[root@controller ~]# kubectl get jobs
NAME                 COMPLETIONS   DURATION   AGE
pre-install-hook-1   1/1           5s         28m
pre-install-hook-2   1/1           5s         28m
pre-install-hook-3   1/1           9s         28m

So all our pre-install jobs were successfully completed.

To verify the execution order we can list the available pods:

[root@controller ~]# kubectl get pods
NAME                       READY   STATUS      RESTARTS   AGE
nginx-statefulset-0        1/1     Running     0          53s
pre-install-hook-1-f2l67   0/1     Completed   0          72s
pre-install-hook-2-hk4xz   0/1     Completed   0          67s
pre-install-hook-3-dpqr5   0/1     Completed   0          62s

Since we only have limited pods, I will just use kubectl describe pod command, which shall give detailed information of all the available pods:

[root@controller ~]# kubectl describe pod | grep -E 'Controlled By:|Started:|Finished:'
Controlled By:  StatefulSet/nginx-statefulset
      Started:      Thu, 18 Mar 2021 23:24:37 +0530
Controlled By:  Job/pre-install-hook-1
      Started:      Thu, 18 Mar 2021 23:24:15 +0530
      Finished:     Thu, 18 Mar 2021 23:24:17 +0530
Controlled By:  Job/pre-install-hook-2
      Started:      Thu, 18 Mar 2021 23:24:19 +0530
      Finished:     Thu, 18 Mar 2021 23:24:22 +0530
Controlled By:  Job/pre-install-hook-3
      Started:      Thu, 18 Mar 2021 23:24:26 +0530
      Finished:     Thu, 18 Mar 2021 23:24:31 +0530

We can ignore the first two lines as that is for the statefulset pod. So as per this data the order of execution was in the following order:

• pre-install job with hook-weight -2
• pre-install job with hook-weight 3
• pre-install job with hook-weight 5

This is the expected behaviour as also informed earlier, the negative hook-weight would be given preference in terms of order of execution.

 

Un-install helm chart

Let me un-install the chart as I would re-create it using the same name in the next example.

[root@controller helm-charts]# helm uninstall hook-weight
release "hook-weight" uninstalled

 

Example-2: Add more jobs using existing helm hooks with different hook-weight

We will use existing chart-5 to add some more pre-install jobs with different hook-weight and observe the behaviour.

 

Create helm hook templates

Following is the content of pre-install-hook-4 with "hook-weight": "10"

 

Following is the content of pre-install-hook-4 with "hook-weight": "-15"

 

Perform pre-checks on the helm chart

Perform helm lint operation to identify any potential issues:

[root@controller helm-charts]# helm lint chart-5/
==> Linting chart-5/
[INFO] Chart.yaml: icon is recommended

1 chart(s) linted, 0 chart(s) failed

Next perform dry run of the helm install command to identify any other issues with our template files (output is trimmed):

[root@controller helm-charts]# helm install hook-weight --dry-run chart-5/
NAME: hook-weight
LAST DEPLOYED: Fri Mar 19 01:17:54 2021
NAMESPACE: default
STATUS: pending-install
REVISION: 1
TEST SUITE: None
HOOKS:
---
# Source: chart-5/templates/pre-install-job-hook-1.yaml
apiVersion: batch/v1
kind: Job
...

 

Install helm chart

Since both lint and dry run execution was success, we can go ahead and install the helm chart:

[root@controller helm-charts]# helm install hook-weight chart-5/
NAME: hook-weight
LAST DEPLOYED: Fri Mar 19 01:06:31 2021
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None

 

Verify helm hook-weight order

Verify the list of jobs:

[root@controller ~]# kubectl get jobs
NAME                 COMPLETIONS   DURATION   AGE
pre-install-hook-1   1/1           4s         13m
pre-install-hook-2   1/1           5s         13m
pre-install-hook-3   1/1           7s         13m
pre-install-hook-4   1/1           7s         13m
pre-install-hook-5   1/1           7s         13m

So all the jobs are in COMPLETED stage, but you may observe the order. You cannot interpret the order of execution using this command as this sorts the output numerically.

Let us verify the pods and use them to check the execution order of our jobs:

[root@controller ~]# kubectl describe pod | grep -E 'Controlled By:|Started:|Finished:'
Controlled By:  StatefulSet/nginx-statefulset
      Started:      Fri, 19 Mar 2021 01:07:06 +0530
Controlled By:  Job/pre-install-hook-1
      Started:      Fri, 19 Mar 2021 01:06:40 +0530
      Finished:     Fri, 19 Mar 2021 01:06:42 +0530
Controlled By:  Job/pre-install-hook-2
      Started:      Fri, 19 Mar 2021 01:06:44 +0530
      Finished:     Fri, 19 Mar 2021 01:06:47 +0530
Controlled By:  Job/pre-install-hook-3
      Started:      Fri, 19 Mar 2021 01:06:49 +0530
      Finished:     Fri, 19 Mar 2021 01:06:54 +0530
Controlled By:  Job/pre-install-hook-4
      Started:      Fri, 19 Mar 2021 01:06:56 +0530
      Finished:     Fri, 19 Mar 2021 01:07:01 +0530
Controlled By:  Job/pre-install-hook-5
      Started:      Fri, 19 Mar 2021 01:06:33 +0530
      Finished:     Fri, 19 Mar 2021 01:06:38 +0530

So you can check the Started and Finished time to get the execution order. Our jobs were executed in the following order:

• pre-install-hook-5 with hook-weight -15, Started at 01:06:33 and Finished at 01:06:38
• pre-install-hook-1 with hook-weight -2, Started at 01:06:40 and Finished at 01:06:42
• pre-install-hook-2 with hook-weight 3, Started at 01:06:44 and Finished at 01:06:47
• pre-install-hook-3 with hook-weight 5, Started at 01:06:49 and Finished at 01:06:54
• pre-install-hook-4 with hook-weight 10, Started at 01:06:56 and Finished at 01:07:01

 

Summary

In this tutorial we learned that Helm hooks provide a way to apply certain Kubernetes resources (usually Jobs) at a particular stage of a deployment, for example, to run a database migration. Hook weight can define the order in which resources should be applied during a deployment, and cause the deployment to halt if something does not succeed.

When Helm evaluates a particular hook, the resources will be sorted by these weights in ascending order. Since annotations can only hold strings, it is important to quote the number used in a hook weight.