How to assign Kubernetes resource quota with examples

Resource quotas allow you to place limits on how many resources a particular namespace can use. Depending on how you have chosen to use namespaces in your organization, they can give you a powerful way to limit the resources that are used by a particular team, application, or group of applications, while still giving developers the freedom to tweak the resource limits of each individual container.

A resource quota, defined by a ResourceQuota object, provides constraints that limit aggregate resource consumption per namespace. It can limit the quantity of objects that can be created in a namespace by type, as well as the total amount of compute resources that may be consumed by resources in that namespace.

Advertisement

Don't get confused between Kubernetes Resource Quota and Resource Limit, The resource quota is applied on the namespace while the resource limit is applied on the containers.

 

Resource quotas for namespaces

  • In Kubernetes, resource quotas are managed by an admission controller.
  • Resource quotas are a useful tool when you want to control the resource costs of different teams or applications, but still want to achieve the utilization benefits of scheduling multiple workloads to the same cluster.
  • If creating or updating a resource violates a quota constraint, the request will fail with HTTP status code 403 FORBIDDEN with a message explaining the constraint that would have been violated.
  • This controller tracks the use of resources such as pods and services, and if a limit is exceeded, it prevents new resources from being created.
  • The resource quota admission controller is configured by one or more ResourceQuota objects created in the namespace.
  • If quota is enabled in a namespace for compute resources like cpu and memory, users must specify requests or limits for those values; otherwise, the quota system may reject pod creation

 

Resource quota types

There are different types of quota we can manage and control. The categories are compute, storage, and objects.

 

Compute resource quota

Compute resources are CPU and memory. For each one, you can specify a limit or request a certain amount. Here is the list of compute-related fields. Note that requests.cpu can be specified as just cpu, and requests.memory can be specified as just memory:

  • limits.cpu: Across all pods in a non-terminal state, the sum of CPU limits cannot exceed this value
  • limits.memory: Across all pods in a non-terminal state, the sum of memory limits cannot exceed this value
  • requests.cpu: Across all pods in a non-terminal state, the sum of CPU requests cannot exceed this value
  • requests.memory: Across all pods in a non-terminal state, the sum of memory requests cannot exceed this value

 

Storage resource quota

The storage resource quota type is a little more complicated. There are two entities you can restrict per namespace: the amount of storage and the number of persistent volume claims. However, in addition to just globally setting the quota on the total storage or the total number of persistent volume claims, you can also do that per storage class. The notation for storage class resource quota is a little verbose, but it gets the job done:

  • requests.storage: The total amount of requested storage across all persistent volume claims
  • persistentvolumeclaims: The maximum number of persistent volume claims allowed in the namespace
  • .storageclass.storage.k8s.io/requests.storage: The total amount of requested storage across all persistent volume claims associated with the storage class name
  • .storageclass.storage.k8s.io/persistentvolumeclaims: The maximum number of persistent volume claims allowed in the namespace that are associated with the storage class name

Kubernetes 1.8 added alpha support for ephemeral storage quotas too:

  • requests.ephemeral-storage: The total amount of requested ephemeral storage across all pods in the namespace claims
  • limits.ephemeral-storage: The total amount of limits for ephemeral storage across all pods in the namespace claims

 

Object count quota

Kubernetes has another category of resource quotas, which is API objects. Since Kubernetes 1.9 you can restrict the number of any namespaced resource (prior to that coverage of objects that can be restricted was a little spotty). The syntax is interesting:

Advertisement
  • count/<resource>.<group> for resources from non-core groups
  • count/<resource> for resources from the core group

Here are some objects you may want to limit (note that deployments can be limited for two separate API groups):

count/persistentvolumeclaims
count/services
count/secrets
count/configmaps
count/replicationcontrollers
count/deployments.apps
count/replicasets.apps
count/statefulsets.apps
count/jobs.batch
count/cronjobs.batch

It is also possible to do generic object count quota on a limited set of resources such as pods, services, secrets, configmaps etc. For example, pods quota counts and enforces a maximum on the number of pods created in a single namespace that are not terminal. You might want to set a pods quota on a namespace to avoid the case where a user creates many small pods and exhausts the cluster's supply of Pod IPs

 

Example: Define CPU quota for a namespace

As quotas will affect all the pods within a namespace, we will start by creating a new namespace using kubectl:

[root@controller ~]# kubectl create namespace quota-example
namespace/quota-example created

Next we will create a YAML file with a CPU quota limit and assign this to the newly created namespace, in this example we have only defined quota limit for CPU for 1 core but you can also add limit for other resource types such as memory, pod count etc.

[root@controller ~]# cat ns-quota-limit.yml
apiVersion: v1
kind: ResourceQuota
metadata:
  name: resource-quota
  namespace: quota-example
spec:
  hard:
    limits.cpu: 1

Next apply this YAML to the newly created NS earlier:

[root@controller ~]# kubectl apply -f ns-quota-limit.yml
resourcequota/resource-quota created

Verify the allocated quota:

[root@controller ~]# kubectl describe ns quota-example
Name:         quota-example
Labels:       <none>
Annotations:  <none>
Status:       Active

Resource Quotas
 Name:       resource-quota
 Resource    Used  Hard
 --------    ---   ---
 limits.cpu  0     1

No LimitRange resource.

Now we will create a simple example pod with nginx image and assign a CPU resource limit of 500m. So with this we should be allowed to create upto two Pods only because after that that CPU quota limit would be reached.

[root@controller ~]# cat pod-nginx-lab-1.yml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: example
  namespace: quota-example
spec:
  selector:
    matchLabels:
      app: example
  template:
    metadata:
    spec:
      containers:
      - name: nginx
        image: nginx
        resources:
          limits:
            cpu: 1100m

We are using Deployment kind so that we can scale the number of Pods to verify our quota limit. Let's create this deployment:

Advertisement
[root@controller ~]# kubectl create -f pod-nginx-lab-1.yml
deployment.apps/example created

Once you have submitted the deployment manifest to Kubernetes with kubectl, check that the pod is running:

[root@controller ~]# kubectl get pods -n quota-example -o wide
NAME                       READY   STATUS              RESTARTS   AGE   IP          NODE                   NOMINATED NODE   READINESS GATES
example-787448d859-5q7dp   0/1     ContainerCreating   0          2s    <none>      worker-2.example.com   <none>           <none>

Now, scale up the deployment and observe that additional pods are created:

[root@controller ~]# kubectl -n quota-example scale deployment/example --replicas=5
deployment.apps/example scaled

Because we specified a CPU limit of 500m, there is no problem scaling our deployment to two replicas, which uses the two cores that we specified in our quota.

[root@controller ~]# kubectl get pods -n quota-example -o wide
NAME                       READY   STATUS    RESTARTS   AGE   IP          NODE                   NOMINATED NODE   READINESS GATES
example-787448d859-5q7dp   1/1     Running   0          31s   10.44.0.2   worker-2.example.com   <none>           <none>
example-787448d859-pk8d5   1/1     Running   0          62s   10.36.0.2   worker-1.example.com   <none>           <none>

But if you now try to scale the deployment so it uses more resources than specified in the quota, you will find that additional pods are not scheduled by Kubernetes:

[root@controller ~]# kubectl -n quota-example get events
....
LAST SEEN   TYPE      REASON              OBJECT                            MESSAGE
4m57s       Warning   FailedCreate        replicaset/example-1-78b64cfbf5   Error creating: pods "example-1-78b64cfbf5-v9qbv" is forbidden: exceeded quota: resource-quota, requested: limits.cpu=600m, used: limits.cpu=500m, limited: limits.cpu=1
4m57s       Warning   FailedCreate        replicaset/example-1-78b64cfbf5   Error creating: pods "example-1-78b64cfbf5-wn52k" is forbidden: exceeded quota: resource-quota, requested: limits.cpu=600m, used: limits.cpu=500m, limited: limits.cpu=1
4m57s       Warning   FailedCreate        replicaset/example-1-78b64cfbf5   Error creating: pods "example-1-78b64cfbf5-kp4gn" is forbidden: exceeded quota: resource-quota, requested: limits.cpu=600m, used: limits.cpu=500m, limited: limits.cpu=1
...

To delete the deployment:

[root@controller ~]# kubectl delete deployments -n quota-example example
deployment.apps "example" deleted

We will also delete the namespace:

[root@controller ~]# kubectl delete ns quota-example
namespace "quota-example" deleted

 

Example: Define a count quota for pods

In this example we will create a namespace with quota limit for the number of pods which can be created within the namespace.

[root@controller ~]# cat pod-quota-limit.yml
apiVersion: v1
kind: ResourceQuota
metadata:
  name: pods-quota
  namespace: pods-quota-ns
spec:
  hard:
    pods: 2

Since we are planning to map this quota to a new namespace, we will first need to create a new namespace:

Advertisement
[root@controller ~]# kubectl create ns pods-quota-ns
namespace/pods-quota-ns created

Now we can create a new quota and apply it to this new created namespace:

[root@controller ~]# kubectl apply -f pod-quota-limit.yml
resourcequota/pods-quota created

Verify the namespace is created successfully:

[root@controller ~]# kubectl get ns pods-quota-ns
NAME            STATUS   AGE
pods-quota-ns   Active   2m6s

To check the resource quota limit which is applied to pods-quota-ns namespace:

[root@controller ~]# kubectl get resourcequota -n pods-quota-ns
NAME         AGE     REQUEST     LIMIT
pods-quota   2m34s   pods: 0/2

We can also use kubectl describe to get more details on the quota limit on respective namespace:

[root@controller ~]# kubectl describe ns pods-quota-ns
Name:         pods-quota-ns
Labels:       <none>
Annotations:  <none>
Status:       Active

Resource Quotas
 Name:     pods-quota
 Resource  Used  Hard
 --------  ---   ---
 pods      0     2

No LimitRange resource.

So, we have defined a hard limit of 2 pods in our pods-quota-ns namespace. Let us create a deployment to make sure, more than 2 pods are not created in this namespace:

[root@controller ~]# cat nginx-example.yml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-1
  namespace: pods-quota-ns
spec:
  selector:
    matchLabels:
      run: nginx
  template:
    metadata:
      labels:
        run: nginx
    spec:
      containers:
      - name: nginx
        image: nginx
        ports:
        - containerPort: 80

Let us create this deployment:

[root@controller ~]# kubectl create -f nginx-example.yml
deployment.apps/nginx-1 created

Our pod has successfully started:

[root@controller ~]# kubectl get pods -n pods-quota-ns
NAME                       READY   STATUS    RESTARTS   AGE
nginx-1-658f4cf99f-j2qkb   1/1     Running   0          23s

Now we will scale the number of pods in our deployment to 5 and check if those are created:

[root@controller ~]# kubectl -n pods-quota-ns scale deployment/nginx-1 --replicas=5
deployment.apps/nginx-1 scaled

List the available pods in pods-quota-ns namespace:

[root@controller ~]# kubectl get pods -n pods-quota-ns
NAME                       READY   STATUS    RESTARTS   AGE
nginx-1-658f4cf99f-j2qkb   1/1     Running   0          18m
nginx-1-658f4cf99f-jhfx2   1/1     Running   0          20s

As you see, even though we gave the scale limit as 5, only 2 pods were created as we have a hard quota limit on number of pods allowed in pods-quota-ns to be maximum as 2.

You can verify the events in this namespace:

[root@controller ~]# kubectl -n pods-quota-ns get events
...
2m59s       Warning   FailedCreate        replicaset/nginx-1-658f4cf99f   Error creating: pods "nginx-1-658f4cf99f-rksfk" is forbidden: exceeded quota: pods-quota, requested: pods=1, used: pods=2, limited: pods=2
2m59s       Warning   FailedCreate        replicaset/nginx-1-658f4cf99f   Error creating: pods "nginx-1-658f4cf99f-567pj" is forbidden: exceeded quota: pods-quota, requested: pods=1, used: pods=2, limited: pods=2
2m59s       Warning   FailedCreate        replicaset/nginx-1-658f4cf99f   Error creating: pods "nginx-1-658f4cf99f-mccth" is forbidden: exceeded quota: pods-quota, requested: pods=1, used: pods=2, limited: pods=2
2m58s       Warning   FailedCreate        replicaset/nginx-1-658f4cf99f   Error creating: pods "nginx-1-658f4cf99f-z79vh" is forbidden: exceeded quota: pods-quota, requested: pods=1, used: pods=2, limited: pods=2
...

The same can be checked using kubectl describe command where the used quota is same as hard limit so no more pods will be allowed to be created in this NS:

[root@controller ~]# kubectl describe ns pods-quota-ns
Name:         pods-quota-ns
Labels:       <none>
Annotations:  <none>
Status:       Active

Resource Quotas
 Name:     pods-quota
 Resource  Used  Hard
 --------  ---   ---
 pods      2     2

No LimitRange resource.

 

Understanding Limit Range

When you are using quotas on a namespace, one requirement is that every container in the namespace must have resource limits and requests defined. Sometimes this requirement can cause complexity and make it more difficult to work quickly with Kubernetes. Kubernetes provides the facility for default requests and limits to be provided at the namespace level. You could use this to provide some sensible defaults to namespaces used by a particular application or team.

We can configure default limits and requests for the containers in a namespace using the LimitRange object. This object allows us to provide defaults for the CPU or memory, or both. If a LimitRange object exists in a namespace, then any container created without the resource requests or limits configured in LimitRange will inherit these values from the limit range.

There are two situations where LimitRange will affect the resource limits or requests when a pod is created:

  • Containers that have no resource limits or requests will inherit the resource limit and requests from the LimitRange object
  • Containers that have no resource limits but do have requests specified will inherit the resource limit from the LimitRange object

If a container already has limits and requests defined, then LimitRange will have no effect. Because containers that specify only limits default the request field to the same value, they will not inherit the request value from LimitRange.

 

Example: Create and apply LimitRange to namespace

In this example we will create a LimitRange and assign it to our existing namespace pods-quota-ns where we had assigned a quota limit.

[root@controller ~]# cat assign-limit-range.yml
apiVersion: v1
kind: LimitRange
metadata:
  name: define-limit
  namespace: pods-quota-ns
spec:
  limits:
  - default:
      memory: 512Mi
      cpu: 1
    defaultRequest:
      memory: 256Mi
      cpu: 500m
    type: Container

Let's create this LimitRange:

[root@controller ~]# kubectl create -f assign-limit-range.yml
limitrange/define-limit created

To get the list of LimitRange in pods-quota-ns namespace:

[root@controller ~]# kubectl get limits -n pods-quota-ns
NAME           CREATED AT
define-limit   2020-11-29T07:15:18Z

We can verify the LimitRange using kubectl describe:

[root@controller ~]# kubectl describe ns pods-quota-ns
Name:         pods-quota-ns
Labels:       <none>
Annotations:  <none>
Status:       Active

Resource Quotas
 Name:     pods-quota
 Resource  Used  Hard
 --------  ---   ---
 pods      2     2

Resource Limits
 Type       Resource  Min  Max  Default Request  Default Limit  Max Limit/Request Ratio
 ----       --------  ---  ---  ---------------  -------------  -----------------------
 Container  cpu       -    -    500m             1              -
 Container  memory    -    -    256Mi            512Mi          -

Instead of using describe with namespace, we can use it with LimitRange directly:

[root@controller ~]# kubectl describe limits -n pods-quota-ns define-limit
Name:       define-limit
Namespace:  pods-quota-ns
Type        Resource  Min  Max  Default Request  Default Limit  Max Limit/Request Ratio
----        --------  ---  ---  ---------------  -------------  -----------------------
Container   cpu       -    -    500m             1              -
Container   memory    -    -    256Mi            512Mi          -

Now let's create a new deployment without specifying the resource limits (but before that I will delete the existing deployment which I created earlier):

[root@controller ~]#  kubectl delete deployment -n pods-quota-ns nginx-1
deployment.apps "nginx-1" deleted

[root@controller ~]# kubectl get pods -n pods-quota-ns
NAME      READY   STATUS    RESTARTS   AGE
nginx-1   1/1     Running   0          25s

Now let's check the details of this pod to make sure if the default resource limit is applied:

[root@controller ~]# kubectl describe pods -n pods-quota-ns nginx-1
Name:         nginx-1
Namespace:    pods-quota-ns
Priority:     0
Node:         worker-1.example.com/192.168.43.49
Start Time:   Sun, 29 Nov 2020 13:03:22 +0530
Labels:       run=nginx-1

...

    State:          Running
      Started:      Sun, 29 Nov 2020 13:03:34 +0530
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     1
      memory:  512Mi
    Requests:
      cpu:        500m
      memory:     256Mi
...

As expected, even when we didn't provided any Resource Limits to this Pod, the pod has default LimitRange applied.

 

To delete the LimitRange we created in pods-quota-ns namespace:

[root@controller ~]# kubectl delete limitranges -n pods-quota-ns define-limit
limitrange "define-limit" deleted

 

Conclusion

In this Kubernetes tutorial we learned how we can define resource quota to namespace to handle resource scarcity across Cluseter nodes. If your namespace has a resource quota, it is helpful to have a default value in place for CPU limit. Here are two of the restrictions that a resource quota imposes on a namespace:

  • Every Container that runs in the namespace must have its own CPU limit.
  • The total amount of CPU used by all Containers in the namespace must not exceed a specified limit.

If a Container does not specify its own CPU limit, it is given the default limit, and then it can be allowed to run in a namespace that is restricted by a quota.

 

Also Read (External):

Configure Minimum and Maximum CPU Constraints for a Namespace
Configure Default CPU Requests and Limits for a Namespace

Didn't find what you were looking for? Perform a quick search across GoLinuxCloud

If my articles on GoLinuxCloud has helped you, kindly consider buying me a coffee as a token of appreciation.

Buy GoLinuxCloud a Coffee

For any other feedbacks or questions you can either use the comments section or contact me form.

Thank You for your support!!

Leave a Comment