How to monitor Kubernetes Pod and Sidecar Container Resources


Kubernetes Tutorial

In my earlier article I had explained about Kubernetes Resources and how to assign resource quota in a namespace. Now let's make things more interesting and learn how to monitor resources of a Kubernetes Pod and Sidecar containers.

Normally in production environments, deployments are integrated with different tools used for resource monitoring such as Prometheus or Grafana but in this tutorial we will focus and plan to use traditional tool i.e. kubectl to monitor Kubernetes pod and container resources LIVE. These methods are not very friendly for production method so I will keep the methods explained in this article for DEV environment for dimensioning use cases such as you wish to figure out the resource required for your application.

You can directly use Kubernetes Metrics API or using kubectl top command which will give you LIVE feed of the current CPU and Memory usage of the Pod. You can simply integrate the kubectl top command with while loop to print and store the resource of the pods at regular intervals with date and time stamp.

We will cover these topics in more detail through out this article.

 

Pre-requisite - Setup Metrics Server

The Kubernetes Metrics Server is a crucial component for monitoring resource usage in a Kubernetes cluster. It collects and aggregates resource usage data, such as CPU and memory consumption, from the kubelets running on each node in the cluster. This data is then exposed via the Metrics API, which can be queried using kubectl top to provide real-time metrics about nodes and pods.

Before installing, ensure that the Metrics Server is not already installed:

kubectl get deployment metrics-server -n kube-system

If the deployment exists, the Metrics Server is already installed.

If the Metrics Server is not installed, you can install it using the following command:

kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

Sample Output:

serviceaccount/metrics-server created
clusterrole.rbac.authorization.k8s.io/system:aggregated-metrics-reader created
clusterrole.rbac.authorization.k8s.io/system:metrics-server created
rolebinding.rbac.authorization.k8s.io/metrics-server-auth-reader created
clusterrolebinding.rbac.authorization.k8s.io/metrics-server:system:auth-delegator created
clusterrolebinding.rbac.authorization.k8s.io/system:metrics-server created
service/metrics-server created
deployment.apps/metrics-server created
apiservice.apiregistration.k8s.io/v1beta1.metrics.k8s.io created

After installation, verify that the Metrics Server is running:

kubectl get deployment metrics-server -n kube-system

Sample Output:

How to monitor Kubernetes Pod and Sidecar Container Resources

Ensure that the Metrics Server is functioning correctly by checking its logs:

kubectl logs -f deployment/metrics-server -n kube-system

In my case I was getting below TLS error which indicates that the Metrics Server is having trouble verifying the TLS certificate of the Kubelet running on the node. This usually happens when the certificate presented by the Kubelet does not include the node's IP address in its Subject Alternative Names (SANs).

E0526 05:47:19.599506       1 scraper.go:149] "Failed to scrape node" err="Get \"https://172.19.200.6:10250/metrics/resource\": tls: failed to verify certificate: x509: cannot validate certificate for 172.19.200.6 because it doesn't contain any IP SANs" node="node01

But since this was test deployment for me so I chose to skip TLS validation by adding the --kubelet-insecure-tls flag:

kubectl edit deployment metrics-server -n kube-system

Add the --kubelet-insecure-tls argument under the args section:

spec:
containers:
- args:
- --cert-dir=/tmp
- --secure-port=10250
- --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
- --kubelet-use-node-status-port
- --kubelet-insecure-tls
...

This argument tells the Metrics Server to skip the certificate verification when scraping the Kubelet metrics endpoint.

NOTE:
This work around to skip TLS validation is not recommended for production.

Save the changes and exit the editor. Kubernetes will automatically restart the Metrics Server with the updated configuration.

After updating the Metrics Server deployment, check the logs again to ensure that it is now able to scrape metrics without the certificate validation errors:

I0526 05:50:13.707602       1 shared_informer.go:311] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0526 05:50:13.708527 1 secure_serving.go:213] Serving securely on [::]:10250
I0526 05:50:13.708698 1 dynamic_serving_content.go:132] "Starting controller" name="serving-cert::/tmp/apiserver.crt::/tmp/apiserver.key"
I0526 05:50:13.708883 1 tlsconfig.go:240] "Starting DynamicServingCertificateController"
I0526 05:50:13.807642 1 shared_informer.go:318] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0526 05:50:13.808149 1 shared_informer.go:318] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0526 05:50:13.808560 1 shared_informer.go:318] Caches are synced for RequestHeaderAuthRequestController

 

Monitoring with kubectl top command

The kubectl top command allows you to view resource usage metrics for nodes and pods in your Kubernetes cluster.

To view resource usage across all nodes in your cluster:

kubectl top nodes

Sample Output:

NAME           CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%   
controlplane 191m 9% 1328Mi 34%
node01 1026m 51% 746Mi 19%
  • CPU(cores): The total CPU usage on the node.
  • CPU%: The percentage of CPU usage relative to the node's total capacity.
  • MEMORY(bytes): The total memory usage on the node.
  • MEMORY%: The percentage of memory usage relative to the node's total capacity.

To view resource usage across all pods in a specific namespace:

kubectl top pods -n <namespace>
How to monitor Kubernetes Pod and Sidecar Container Resources

To view resource usage for specific containers within a pod use --containers argument:

kubectl top pod <pod-name> -n <namespace> --containers
How to monitor Kubernetes Pod and Sidecar Container Resources

This provides a detailed view of resource usage for each container within the specified pod.

 

Monitoring using kubectl proxy and Metrics API

Open a terminal on your local machine (or any machine with kubectl configured to access your Kubernetes cluster) and run:

kubectl proxy

This command starts a local proxy to the Kubernetes API server and listens on localhost:8001.

In another terminal (while kubectl proxy is running), run the curl command to access the Metrics API:

curl -s http://localhost:8001/apis/metrics.k8s.io/v1beta1/namespaces/<namespace>/pods/<pod-name> | jq '.containers[] | select(.name=="<container-name>") | {name: .name, usage: .usage}'

For Example to check the resource of main-container inside demo namespace part of demo-pod:

curl -s http://localhost:8001/apis/metrics.k8s.io/v1beta1/namespaces/demo/pods/demo-pod | jq '.containers[] | select(.name=="main-container") | {name: .name, usage: .usage}'
How to monitor Kubernetes Pod and Sidecar Container Resources

Similarly to check resource usage of sidecar-container-1:

curl -s http://localhost:8001/apis/metrics.k8s.io/v1beta1/namespaces/demo/pods/demo-pod | jq '.containers[] | select(.name=="sidecar-container-1") | {name: .name, usage: .usage}'
How to monitor Kubernetes Pod and Sidecar Container Resources

The Metrics API returns CPU usage in nanocores (n) and memory usage in kibibytes (Ki). You can convert these units to more familiar terms like millicores (m) for CPU and megabytes (MB) or gigabytes (GB) for memory.

# Assuming the following output from the previous step
cpu_nanocores=246694653
memory_kib=12236

# Convert CPU usage to millicores
cpu_millicores=$((cpu_nanocores / 1000000))

# Convert memory usage to MiB
memory_mib=$((memory_kib / 1024))

echo "CPU usage: ${cpu_millicores}m"
echo "Memory usage: ${memory_mib}Mi"

This script converts and prints the CPU usage in millicores and memory usage in MiB.

CPU usage: 246m
Memory usage: 11Mi

 

Script to monitor Kubernetes Resources at regular interval

Here is a simple shell script which can be used to monitor resources of all pods within a namespace at regular intervals and store the output in a log file:

#!/bin/bash

NAMESPACE="default"
OUTPUT_FILE="resource_usage.log"
SLEEP_INTERVAL=300  # Interval in seconds (300 seconds = 5 minutes)

# Create output file with headers if it doesn't exist
if [ ! -f $OUTPUT_FILE ]; then
    echo "timestamp,pod_name,container_name,cpu_usage,memory_usage" > $OUTPUT_FILE
fi

while true; do
    # Get current timestamp
    timestamp=$(date +%Y-%m-%dT%H:%M:%S%z)

    # Get resource usage for all pods in the specified namespace
    kubectl top pod -n $NAMESPACE --containers | awk -v timestamp="$timestamp" 'NR>1 {print timestamp "," $1 "," $2 "," $3 "," $4}' >> $OUTPUT_FILE

    # Sleep for the specified interval before the next iteration
    sleep $SLEEP_INTERVAL
done
  • NAMESPACE: The Kubernetes namespace from which to collect metrics.
  • OUTPUT_FILE: The file where resource usage data will be logged.
  • SLEEP_INTERVAL: The interval in seconds between data collection points (set to 300 seconds, or 5 minutes).
  • Headers: The script initializes the log file with headers if it doesn't already exist.
  • Infinite Loop: The while true loop ensures the script runs indefinitely, collecting data at regular intervals.
  • Timestamp: The current timestamp is collected at each interval.
  • Data Collection: kubectl top pod -n $NAMESPACE --containers collects resource usage data for all containers within the pods in the specified namespace.
  • Data Logging: The data is formatted and appended to the log file.

This will output in below format:

How to monitor Kubernetes Pod and Sidecar Container Resources

Alternatively we can also use Metrics API in a shell script to monitor resource usage at regular intervals:

#!/bin/bash

NAMESPACE="restruct-2240-0"
POD_NAME="demo-pod"
OUTPUT_FILE="specific_pod_metrics_usage.log"
SLEEP_INTERVAL=300  # Interval in seconds (300 seconds = 5 minutes)

# Start kubectl proxy in the background
kubectl proxy &
PROXY_PID=$!

# Ensure the proxy is terminated when the script exits
trap "kill $PROXY_PID" EXIT

# Add a brief delay to ensure kubectl proxy starts
sleep 5

# Create output file with headers if it doesn't exist
if [ ! -f $OUTPUT_FILE ]; then
    echo "timestamp,pod_name,container_name,cpu_usage(millicores),memory_usage(Mi)" > $OUTPUT_FILE
fi

while true; do
    # Get current timestamp
    timestamp=$(date +%Y-%m-%dT%H:%M:%S%z)

    # Fetch metrics from the Metrics API for the specific pod
    response=$(curl -s http://localhost:8001/apis/metrics.k8s.io/v1beta1/namespaces/$NAMESPACE/pods/$POD_NAME)

    # Check if the response is valid
    if [ -n "$response" ] && echo "$response" | jq -e . > /dev/null 2>&1; then
        # Parse the response and append to the output file
        echo "$response" | jq -r --arg timestamp "$timestamp" --arg pod_name "$POD_NAME" '
        .containers[] | 
        .name as $container_name | 
        (.usage.cpu // "0n" | sub("n"; "") | tonumber / 1000000 | tostring + "m") as $cpu_usage_millicores | 
        (.usage.memory // "0Ki" | sub("Ki"; "") | tonumber / 1024 | tostring + "Mi") as $memory_usage_mi |
        "\($timestamp),\($pod_name),\($container_name),\($cpu_usage_millicores),\($memory_usage_mi)"
        ' >> $OUTPUT_FILE
    else
        echo "$timestamp,$POD_NAME,Error fetching metrics or invalid JSON response" >> $OUTPUT_FILE
    fi

    # Sleep for the specified interval before the next iteration
    sleep $SLEEP_INTERVAL
done

This jq command parses the JSON response to extract container names and their CPU and memory usage:

  • .containers[]: Iterates over each container in the pod.
  • .name as $container_name: Extracts the container name.
  • (.usage.cpu // "0n" | sub("n"; "") | tonumber / 1000000 | tostring + "m") as $cpu_usage_millicores: Converts CPU usage from nanocores to millicores and appends "m".
  • (.usage.memory // "0Ki" | sub("Ki"; "") | tonumber / 1024 | tostring + "Mi") as $memory_usage_mi: Converts memory usage from kibibytes to mebibytes and appends "Mi".
  • \($timestamp),\($pod_name),\($container_name),\($cpu_usage_millicores),\($memory_usage_mi): Formats the output string with the extracted values and appends it to the log file.

Sample Output:

How to monitor Kubernetes Pod and Sidecar Container Resources

Well by the look of it we can assume that using Metrics API in a script can be tricky considering the fact that the resource information is not in user friendly format. The CPU usage is given in nanocores (n) while Memory usage is given in kibibytes (Ki) so we had to add additional logic to convert them to millicore (m) or cores for CPU and MB or GB for Memory.

Now it depends on your preference and requirement to use respective method to monitor resources of Kubernetes Pod and Sidecar Containers.

But as recommended earlier for production environment you should use tools such as Prometheus and Grafana for resource monitoring.

 

Views: 115
Deepak Prasad

Deepak Prasad

He is the founder of GoLinuxCloud and brings over a decade of expertise in Linux, Python, Go, Laravel, DevOps, Kubernetes, Git, Shell scripting, OpenShift, AWS, Networking, and Security. With extensive experience, he excels in various domains, from development to DevOps, Networking, and Security, ensuring robust and efficient solutions for diverse projects. You can connect with him on his LinkedIn profile.

Can't find what you're searching for? Let us assist you.

Enter your query below, and we'll provide instant results tailored to your needs.

If my articles on GoLinuxCloud has helped you, kindly consider buying me a coffee as a token of appreciation.

Buy GoLinuxCloud a Coffee

For any other feedbacks or questions you can send mail to admin@golinuxcloud.com

Thank You for your support!!

Leave a Comment