Table of Contents
Kubernetes provides three layers of health checking. First, in the form of HTTP or TCP checks, K8s can attempt to connect to a particular endpoint and give a status of healthy on a successful connection. Second, application-specific health checks can be performed using command-line scripts. We can also use the exec container to run a health check from within your container. Anything that exits with a 0 status will be considered healthy. We will cover them in more details and with different examples and demos in this tutorial.
Using Probes for Kubernetes Health Checks
Kubernetes has a concept of probes to provide the seam between the orchestration engine and the application developer. Kubernetes uses these probes to find out more about the inner state of the application service at hand. Probes are part of container spec and are executed locally, inside each container.
There are three probes available with Kubernetes used to perform health checks:
- Liveness probe: used to continue checking the availability of a Pod
- Readiness Probe: used to make sure a Pod is not published as available until the readinessProbe has been able to access it.
- Startup Probe: If we define a startup probe for a container, then Kubernetes does not execute the liveness or readiness probes, as long as the container's startup probe does not succeed.
These probes further utilize different probes to get the health of the application inside the container:
- tcpSocket: Just check that TCP socket for a port is successful
- exec: Run a command that returns 0 for success
- httpGet: an HTTP request returns a response code between 200 and 399
You can further utilise different options in these probes spec to control the behaviour of liveness and readiness probes:
initialDelaySeconds: Number of seconds after the container has started before liveness or readiness probes are initiated. Defaults to 0 seconds. Minimum value is 0.
periodSeconds: How often (in seconds) to perform the probe. Default to 10 seconds. Minimum value is 1.
timeoutSeconds: Number of seconds after which the probe times out. Defaults to 1 second. Minimum value is 1.
successThreshold: Minimum consecutive successes for the probe to be considered successful after having failed. Defaults to 1. Must be 1 for liveness and startup Probes. Minimum value is 1.
failureThreshold: When a probe fails, Kubernetes will try
failureThresholdtimes before giving up. Giving up in case of liveness probe means restarting the container. In case of readiness probe the Pod will be marked Unready. Defaults to 3. Minimum value is 1.
Kubernetes liveness probe
Kubectl watches over your containers. If a container process crashes,
kubelet will take care of it based on the restart policy. But this is not always enough. Your process may not crash, but instead run into an infinite loop or a deadlock. The restart policy might not be nuanced enough. With a liveness probe, you get to decide when a container is considered alive.
Kubernetes uses the liveness probe to decide when a container needs to be killed and when another instance should be launched instead. Since Kubernetes operates at a pod level, the respective pod is killed if at least one of its containers reports as being unhealthy
Alternatively, we can say it the other way around: only if all the containers of a pod report to be healthy, is the pod considered to be healthy.
Some guidelines on using liveness probe to perform health check in Kubernetes:
- Liveness probes should be used with caution. The wrong configuration of this probe can result in cascading failures in your services and container restart loops. As a quick experiment, you can redeploy the voting application manifest where you replace the readiness probe with a liveness probe, with similar configuration but very short timeouts and delays—you will experience multiple random crashes and poor availability of the application!
- Do not use liveness probes unless you have a good reason for this. A good reason may, for example, be a known issue with a deadlock in your application that has an as yet unknown root cause.
- Execute simple and fast checks that determine the status of the process, not its dependencies. In other words, you do not want to check external dependencies' statuses in the liveness probe—this can lead to cascading failures due to an avalanche of container restarts and overloading a small subset of Service Pods.
- If your process running in the container is able to crash or exit whenever it encounters an unrecoverable error, you probably do not need a liveness probe at all.
- Use conservative settings for
initialDelaySecondsto avoid any premature container restarts and falling into a restart loop.
Example-1: Define liveness probe with exec
In this example we will create an nginx container using a simple Pod and use
exec probe to check for nginx service status. This means that as long as nginx service is in '
running' state, the Pod will be considered alove or else it will be killed based on other configuration options such as
failureThreshold, number of retries etc.
Following is the YAML file for my Pod with liveness probe defined:
[root@controller ~]# cat liveness-eg-1.yaml apiVersion: v1 kind: Pod metadata: name: liveness-demo namespace: default spec: containers: - name: test-liveness-probe image: nginx ports: - containerport: 80 livenessProbe: exec: command: - sh - -c - service nginx status | grep running initialDelaySeconds: 10 periodSeconds: 5
The relevant part is in the
livenessProbe section. First, we define a command that Kubernetes will execute as a probe inside the container. In our case, we have a nginx container and use exec probe to execute "
service nginx status" command and grep for the string '
running'. By default the nginx service is expected to be in running state.
The other two settings,
periodSeconds, define how long Kubernetes should wait after starting the container until it first executes the probe and how frequently the probe should be executed thereafter. In our case, Kubernetes waits for 10 seconds prior to executing the first probe and then executes a probe every 5 seconds.
We will create this pod and check the status of the Pod:
[root@controller ~]# kubectl create -f liveness-eg-1.yaml pod/liveness-demo created
Check the status of the Pod:
So at the moment our Pod is in running state which would mean that liveness probe is able to find the nginx service running. We can also use
kubectl describe command to get more details.
For the demonstration purpose, I will intentionally fail the liveness probe by setting incorrect command and will create another pod by the name
fail-liveness-demo. Here even though I have intentionally given a wrong command, the pod seems to be in
[root@controller ~]# kubectl get pods NAME READY STATUS RESTARTS AGE fail-liveness-demo 1/1 Running 0 23s liveness-demo 1/1 Running 0 4m11s
So let's use
kubectl describe pod fail-liveness-demo command to get the details:
Events, we can see that the Liveness Probe has failed but by default the Pod will attempt to restart the container 3 times which is the default failure threshold after which the container will be killed.
We can check the status of the pods in few seconds:
[root@controller ~]# kubectl get pods NAME READY STATUS RESTARTS AGE fail-liveness-demo 0/1 CrashLoopBackOff 3 4m41s liveness-demo 1/1 Running 0 8m29s
As expected, our
fail-liveness-demo has been marked as failed. You can get more details using
kubectl describe pod fail-liveness-demo command:
Let's delete both these pods:
[root@controller ~]# kubectl delete pod liveness-demo pod "liveness-demo" deleted [root@controller ~]# kubectl delete pod fail-liveness-demo pod "fail-liveness-demo" deleted
Example-2: Define liveness probe with httpGet
In this example we will use
httpGet to probe the liveness of a pod. Normally for containers with webservers we can also directly use the kubelet to send an HTTP GET request to the server that is running in the container and listening on port 80. If the handler for the server's
/ path returns a success code, the kubelet considers the container to be alive and healthy. If the handler returns a failure code, the kubelet kills the container and restarts it.
Here is my sample YAML file to create a Pod with liveness probe using
[root@controller ~]# cat liveness-eg-3.yaml apiVersion: v1 kind: Pod metadata: name: liveness-probe-httpget namespace: default spec: containers: - name: test-liveness-probe image: nginx ports: - containerport: 80 livenessProbe: httpGet: path: / port: 8080 initialDelaySeconds: 10 periodSeconds: 5
Create the pod and check the status:
[root@controller ~]# kubectl create -f liveness-eg-3.yaml pod/liveness-probe-httpget created [root@controller ~]# kubectl get pods NAME READY STATUS RESTARTS AGE liveness-probe-httpget 1/1 Running 0 7s
So our pod is in running state, we will also check the events of this pod using
kubectl describe command:
[root@controller ~]# kubectl describe pods liveness-probe-httpget
Again for the demonstration purpose I intentionally gave a wrong port to check the liveness probe in another pod and here is the sample Event from that Pod:
Warning Unhealthy 1s kubelet Liveness probe failed: Get "http://10.36.0.2:8080/": dial tcp 10.36.0.2:8080: connect: connection refused
and the respective pod has entered into
CrashLoopBackOff state with continuous restarts:
[root@controller ~]# kubectl get pods NAME READY STATUS RESTARTS AGE liveness-probe-httpget 1/1 Running 0 7m21s liveness-probe-httpget-fail 0/1 CrashLoopBackOff 6 6m26s
Let's delete both these pods:
[root@controller ~]# kubectl delete pod liveness-probe-httpget liveness-probe-httpget-fail pod "liveness-probe-httpget" deleted pod "liveness-probe-httpget-fail" deleted
Example-3: Define liveness probe with tcpSocket
Kubernetes also supports health checks via simple TCP socket checks. With this configuration, the
kubelet will attempt to open a socket to your container on the specified port. If it can establish a connection, the container is considered healthy, if it can't it is considered a failure.
Following is the YAML file to create this Pod with
tcpSocket as the probe for liveness health check:
[root@controller ~]# cat liveness-eg-4.yaml apiVersion: v1 kind: Pod metadata: name: liveness-probe-tcpsocket namespace: default spec: containers: - name: test-liveness-probe image: nginx ports: - containerport: 80 livenessProbe: tcpSocket: port: 80 initialDelaySeconds: 10 periodSeconds: 5
Create this pod and check the status:
[root@controller ~]# kubectl create -f liveness-eg-4.yaml pod/liveness-probe-tcpsocket created [root@controller ~]# kubectl get pods NAME READY STATUS RESTARTS AGE liveness-probe-tcpsocket 1/1 Running 0 99s
The pod seems to be in running state but we must also check the Events to make sure there are no liveness probe failures:
[root@controller ~]# kubectl describe pods liveness-probe-tcpsocket
Sample Events output from my controller node for this command:
Let's delete this pod:
[root@controller ~]# kubectl delete pod liveness-probe-tcpsocket pod "liveness-probe-tcpsocket" deleted
Kubernetes readiness probe
Kubernetes uses a readiness probe to decide when a service instance, that is, a container, is ready to accept traffic. Pods that are not ready (a Pod is ready only if all of its containers are considered ready) will be removed from the Service Endpoints list until they become ready again. In other words, it is a signal for notifying that a given Pod can be used for requests incoming to the Service.
Some guidelines which are recommended to be followed when using readiness probe to perform health checks in Kubernetes:
- Use this probe whenever your containers may not be ready to properly serve traffic as soon as the container is started.
- Ensure that you check the cache warm-up or database migration status during readiness probe evaluation. You may also consider starting the actual process of a warm-up if it hasn't been started yet, but use this approach with caution—a readiness probe will be executed constantly throughout the life cycle of a Pod, which means you shouldn't do any costly operations for every request. Alternatively, you may want to use a startup probe for this purpose, newly-introduced in Kubernetes 1.16.
- For microservice applications that expose HTTP endpoints, consider always configuring the httpGet readiness probe. This will ensure that all cases are covered when a container is successfully running but the HTTP server is not fully initialized.
- It is a good idea to use a separate, dedicated HTTP endpoint for readiness checks in your application, for example, a common convention is using
- If you are checking the state of dependencies (external database and logging services) in this type of probe, be careful with shared dependencies, such as SQL Server in the voting application. In this case, you should consider using a probe timeout, which is greater than the maximum allowed timeout for the external dependency— otherwise, you may get cascading failures and lower availability instead of occasionally increased latency.
Example: Define readiness probe with httpget
In this example we will use readiness probe to determine the health of the container in an deployment using
httpget. Following is ,y sample YAML file to create a deployment:
[root@controller ~]# cat readiness-eg-1.yaml apiVersion: apps/v1 kind: Deployment metadata: labels: type: dev name: readiness-probe-demo spec: replicas: 2 selector: matchLabels: type: dev template: metadata: labels: type: dev spec: containers: - image: nginx name: nginx readinessProbe: httpGet: path: / port: 80 initialDelaySeconds: 30 periodSeconds: 10 timeoutSeconds: 1 successThreshold: 1 failureThreshold: 3
Let's create this deployment and check the status of pods created as part of this deployment:
[root@controller ~]# kubectl create -f readiness-eg-1.yaml deployment.apps/readiness-probe-demo created
Check the status of the pods:
As you can see the containers are running but are still not marked as "
READY". It is because we had defined
initialDelaySeconds as 30 to give some time for the nginx service to come up before we perform the health check.
We check the status again in few seconds and now the pods are marked as
You can use
kubectl describe command to check the Events for any failures.
[root@controller ~]# kubectl describe pod readiness-probe-demo-65b4cd6f7f-47rht
Let's delete our deployment
[root@controller ~]# kubectl delete deployment readiness-probe-demo deployment.apps "readiness-probe-demo" deleted
Kubernetes startup probe
Startup probes have been introduced in Kubernetes 1.16 to support cases when a container may require more time for initialization than
periodSeconds set in the readiness probe. In general, you should use the same handler configuration for startup probes that you would for readiness probes but use larger delays. If a container is not ready within
initialDelaySeconds + failureThreshold * periodSeconds for a readiness probe, then the container will be killed and subject to the Pod's restart policy.
If we define a startup probe for a container, then Kubernetes does not execute the liveness or readiness probes, as long as the container's startup probe does not succeed. Once again, Kubernetes looks at pods and starts executing liveness and readiness probes on its containers if the startup probes of all the pod's containers succeed.
When should you use startup probes?
There might be situations where we have to account for exceptionally long startup and initialization times, such as when containerizing a legacy application. We could technically configure the readiness or the liveness probes to account for this fact, but that would defeat the purpose of these probes. The latter probes are meant to provide quick feedback to Kubernetes on the health and availability of the container. If we configure for long initial delays or periods, then this would counter the desired outcome.
Example: Using startup probe with httpget
We will use one of our previous examples to demonstrate the behaviour of starup probes. We will create a simple pod using the following YAML file:
[root@controller ~]# cat startup-eg-1.yaml apiVersion: v1 kind: Pod metadata: name: startup-probe-httpget namespace: default spec: containers: - name: nginx image: nginx ports: - containerport: 80 startupProbe: httpGet: path: / port: 80 initialDelaySeconds: 10 failureThreshold: 30 periodSeconds: 5
As you can see, the syntax to use startup probe is same as liveness and readiness probe. We just need to change the probe name.
Let us create this pod and check the status:
[root@controller ~]# kubectl create -f startup-eg-1.yaml pod/startup-probe-httpget created [root@controller ~]# kubectl get pods NAME READY STATUS RESTARTS AGE startup-probe-httpget 0/1 Running 0 16s
So our pod with startup probe was created and is in
Running status but is yet not marked as
READY. After waiting for few seconds and now the pod is marked as running:
[root@controller ~]# kubectl get pods NAME READY STATUS RESTARTS AGE startup-probe-httpget 1/1 Running 0 3m42s
So the only advantage we get here is that the application will have a maximum of around 3.3 minutes (10 + 30 * 5 = 200s) to finish its startup. To demonstrate this I have created another pod with incorrect port value for the health check with startup probe.
Warning Unhealthy 2m44s (x21 over 4m24s) kubelet Startup probe failed: Get "http://10.36.0.2:8080/": dial tcp 10.36.0.2:8080: connect: connection refused
Here this pod has been marked unhealthy for over 4m24s and yet it has been restarted only once because the probe waited for 3.3 minutes before restarting the container.
[root@controller ~]# kubectl get pods NAME READY STATUS RESTARTS AGE startup-probe-httpget 1/1 Running 0 8m20s startup-probe-httpget-fail 0/1 Running 1 4m41s
Let's delete these pods:
[root@controller ~]# kubectl delete pod startup-probe-httpget startup-probe-httpget-fail pod "startup-probe-httpget" deleted pod "startup-probe-httpget-fail" deleted
In this tutorial we learned about different types of probes available in Kubernetes which can be used to check the health of Pods and containers. You may choose the type of probe based on your requirement. But liveness probes are not used very frequently unless you expect your application to be stuck in some deadlock scenario. Most preferred method would be readiness probe in most cases while for legacy applications which takes more time to come up, we can use startup probes. The usage syntax for startup probe is exactly same as liveness and readiness probe. You can further use exec, httpget or tcpsocket with any of these probes to perform the health check.
Related Searches: health check in kubernetes, k8s health check, kubernetes disable health check, health check kubernetes, kubernetes health check https, kubernetes pod health check, kubernetes health check deployment