Resource quotas allow you to place limits on how many resources a particular namespace can use. Depending on how you have chosen to use namespaces in your organization, they can give you a powerful way to limit the resources that are used by a particular team, application, or group of applications, while still giving developers the freedom to tweak the resource limits of each individual container.
A resource quota, defined by a ResourceQuota object, provides constraints that limit aggregate resource consumption per namespace. It can limit the quantity of objects that can be created in a namespace by type, as well as the total amount of compute resources that may be consumed by resources in that namespace.
Don't get confused between Kubernetes Resource Quota and Resource Limit, The resource quota is applied on the namespace while the resource limit is applied on the containers.
Resource quotas for namespaces
- In Kubernetes, resource quotas are managed by an admission controller.
- Resource quotas are a useful tool when you want to control the resource costs of different teams or applications, but still want to achieve the utilization benefits of scheduling multiple workloads to the same cluster.
- If creating or updating a resource violates a quota constraint, the request will fail with HTTP status code 403 FORBIDDEN with a message explaining the constraint that would have been violated.
- This controller tracks the use of resources such as pods and services, and if a limit is exceeded, it prevents new resources from being created.
- The resource quota admission controller is configured by one or more ResourceQuota objects created in the namespace.
- If quota is enabled in a namespace for compute resources like cpu and memory, users must specify requests or limits for those values; otherwise, the quota system may reject pod creation
Resource quota types
There are different types of quota we can manage and control. The categories are compute, storage, and objects.
Compute resource quota
Compute resources are CPU and memory. For each one, you can specify a limit or request a certain amount. Here is the list of compute-related fields. Note that requests.cpu can be specified as just cpu, and requests.memory can be specified as just memory:
limits.cpu: Across all pods in a non-terminal state, the sum of CPU limits cannot exceed this valuelimits.memory: Across all pods in a non-terminal state, the sum of memory limits cannot exceed this valuerequests.cpu: Across all pods in a non-terminal state, the sum of CPU requests cannot exceed this valuerequests.memory: Across all pods in a non-terminal state, the sum of memory requests cannot exceed this value
Storage resource quota
The storage resource quota type is a little more complicated. There are two entities you can restrict per namespace: the amount of storage and the number of persistent volume claims. However, in addition to just globally setting the quota on the total storage or the total number of persistent volume claims, you can also do that per storage class. The notation for storage class resource quota is a little verbose, but it gets the job done:
requests.storage: The total amount of requested storage across all persistent volume claimspersistentvolumeclaims: The maximum number of persistent volume claims allowed in the namespace.storageclass.storage.k8s.io/requests.storage: The total amount of requested storage across all persistent volume claims associated with the storage class name.storageclass.storage.k8s.io/persistentvolumeclaims: The maximum number of persistent volume claims allowed in the namespace that are associated with the storage class name
Kubernetes 1.8 added alpha support for ephemeral storage quotas too:
requests.ephemeral-storage: The total amount of requested ephemeral storage across all pods in the namespace claimslimits.ephemeral-storage: The total amount of limits for ephemeral storage across all pods in the namespace claims
Object count quota
Kubernetes has another category of resource quotas, which is API objects. Since Kubernetes 1.9 you can restrict the number of any namespaced resource (prior to that coverage of objects that can be restricted was a little spotty). The syntax is interesting:
count/<resource>.<group>for resources from non-core groupscount/<resource>for resources from the core group
Here are some objects you may want to limit (note that deployments can be limited for two separate API groups):
count/persistentvolumeclaims count/services count/secrets count/configmaps count/replicationcontrollers count/deployments.apps count/replicasets.apps count/statefulsets.apps count/jobs.batch count/cronjobs.batch
It is also possible to do generic object count quota on a limited set of resources such as pods, services, secrets, configmaps etc. For example, pods quota counts and enforces a maximum on the number of pods created in a single namespace that are not terminal. You might want to set a pods quota on a namespace to avoid the case where a user creates many small pods and exhausts the cluster's supply of Pod IPs
Example: Define CPU quota for a namespace
As quotas will affect all the pods within a namespace, we will start by creating a new namespace using kubectl:
[root@controller ~]# kubectl create namespace quota-example
namespace/quota-example created
Next we will create a YAML file with a CPU quota limit and assign this to the newly created namespace, in this example we have only defined quota limit for CPU for 1 core but you can also add limit for other resource types such as memory, pod count etc.
[root@controller ~]# cat ns-quota-limit.yml apiVersion: v1 kind: ResourceQuota metadata: name: resource-quota namespace: quota-example spec: hard: limits.cpu: 1
Next apply this YAML to the newly created NS earlier:
[root@controller ~]# kubectl apply -f ns-quota-limit.yml
resourcequota/resource-quota created
Verify the allocated quota:
[root@controller ~]# kubectl describe ns quota-example
Name: quota-example
Labels: <none>
Annotations: <none>
Status: Active
Resource Quotas
Name: resource-quota
Resource Used Hard
-------- --- ---
limits.cpu 0 1
No LimitRange resource.
Now we will create a simple example pod with nginx image and assign a CPU resource limit of 500m. So with this we should be allowed to create upto two Pods only because after that that CPU quota limit would be reached.
[root@controller ~]# cat pod-nginx-lab-1.yml apiVersion: apps/v1 kind: Deployment metadata: name: example namespace: quota-example spec: selector: matchLabels: app: example template: metadata: spec: containers: - name: nginx image: nginx resources: limits: cpu: 1100m
We are using Deployment kind so that we can scale the number of Pods to verify our quota limit. Let's create this deployment:
[root@controller ~]# kubectl create -f pod-nginx-lab-1.yml
deployment.apps/example created
Once you have submitted the deployment manifest to Kubernetes with kubectl, check that the pod is running:
[root@controller ~]# kubectl get pods -n quota-example -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES example-787448d859-5q7dp 0/1 ContainerCreating 0 2s <none> worker-2.example.com <none> <none>
Now, scale up the deployment and observe that additional pods are created:
[root@controller ~]# kubectl -n quota-example scale deployment/example --replicas=5
deployment.apps/example scaled
Because we specified a CPU limit of 500m, there is no problem scaling our deployment to two replicas, which uses the two cores that we specified in our quota.
[root@controller ~]# kubectl get pods -n quota-example -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES example-787448d859-5q7dp 1/1 Running 0 31s 10.44.0.2 worker-2.example.com <none> <none> example-787448d859-pk8d5 1/1 Running 0 62s 10.36.0.2 worker-1.example.com <none> <none>
But if you now try to scale the deployment so it uses more resources than specified in the quota, you will find that additional pods are not scheduled by Kubernetes:
[root@controller ~]# kubectl -n quota-example get events .... LAST SEEN TYPE REASON OBJECT MESSAGE 4m57s Warning FailedCreate replicaset/example-1-78b64cfbf5 Error creating: pods "example-1-78b64cfbf5-v9qbv" is forbidden: exceeded quota: resource-quota, requested: limits.cpu=600m, used: limits.cpu=500m, limited: limits.cpu=1 4m57s Warning FailedCreate replicaset/example-1-78b64cfbf5 Error creating: pods "example-1-78b64cfbf5-wn52k" is forbidden: exceeded quota: resource-quota, requested: limits.cpu=600m, used: limits.cpu=500m, limited: limits.cpu=1 4m57s Warning FailedCreate replicaset/example-1-78b64cfbf5 Error creating: pods "example-1-78b64cfbf5-kp4gn" is forbidden: exceeded quota: resource-quota, requested: limits.cpu=600m, used: limits.cpu=500m, limited: limits.cpu=1 ...
To delete the deployment:
[root@controller ~]# kubectl delete deployments -n quota-example example
deployment.apps "example" deleted
We will also delete the namespace:
[root@controller ~]# kubectl delete ns quota-example
namespace "quota-example" deleted
Example: Define a count quota for pods
In this example we will create a namespace with quota limit for the number of pods which can be created within the namespace.
[root@controller ~]# cat pod-quota-limit.yml apiVersion: v1 kind: ResourceQuota metadata: name: pods-quota namespace: pods-quota-ns spec: hard: pods: 2
Since we are planning to map this quota to a new namespace, we will first need to create a new namespace:
[root@controller ~]# kubectl create ns pods-quota-ns
namespace/pods-quota-ns created
Now we can create a new quota and apply it to this new created namespace:
[root@controller ~]# kubectl apply -f pod-quota-limit.yml
resourcequota/pods-quota created
Verify the namespace is created successfully:
[root@controller ~]# kubectl get ns pods-quota-ns NAME STATUS AGE pods-quota-ns Active 2m6s
To check the resource quota limit which is applied to pods-quota-ns namespace:
[root@controller ~]# kubectl get resourcequota -n pods-quota-ns NAME AGE REQUEST LIMIT pods-quota 2m34s pods: 0/2
We can also use kubectl describe to get more details on the quota limit on respective namespace:
[root@controller ~]# kubectl describe ns pods-quota-ns
Name: pods-quota-ns
Labels: <none>
Annotations: <none>
Status: Active
Resource Quotas
Name: pods-quota
Resource Used Hard
-------- --- ---
pods 0 2
No LimitRange resource.
So, we have defined a hard limit of 2 pods in our pods-quota-ns namespace. Let us create a deployment to make sure, more than 2 pods are not created in this namespace:
[root@controller ~]# cat nginx-example.yml apiVersion: apps/v1 kind: Deployment metadata: name: nginx-1 namespace: pods-quota-ns spec: selector: matchLabels: run: nginx template: metadata: labels: run: nginx spec: containers: - name: nginx image: nginx ports: - containerPort: 80
Let us create this deployment:
[root@controller ~]# kubectl create -f nginx-example.yml
deployment.apps/nginx-1 created
Our pod has successfully started:
[root@controller ~]# kubectl get pods -n pods-quota-ns NAME READY STATUS RESTARTS AGE nginx-1-658f4cf99f-j2qkb 1/1 Running 0 23s
Now we will scale the number of pods in our deployment to 5 and check if those are created:
[root@controller ~]# kubectl -n pods-quota-ns scale deployment/nginx-1 --replicas=5
deployment.apps/nginx-1 scaled
List the available pods in pods-quota-ns namespace:
[root@controller ~]# kubectl get pods -n pods-quota-ns NAME READY STATUS RESTARTS AGE nginx-1-658f4cf99f-j2qkb 1/1 Running 0 18m nginx-1-658f4cf99f-jhfx2 1/1 Running 0 20s
As you see, even though we gave the scale limit as 5, only 2 pods were created as we have a hard quota limit on number of pods allowed in pods-quota-ns to be maximum as 2.
You can verify the events in this namespace:
[root@controller ~]# kubectl -n pods-quota-ns get events ... 2m59s Warning FailedCreate replicaset/nginx-1-658f4cf99f Error creating: pods "nginx-1-658f4cf99f-rksfk" is forbidden: exceeded quota: pods-quota, requested: pods=1, used: pods=2, limited: pods=2 2m59s Warning FailedCreate replicaset/nginx-1-658f4cf99f Error creating: pods "nginx-1-658f4cf99f-567pj" is forbidden: exceeded quota: pods-quota, requested: pods=1, used: pods=2, limited: pods=2 2m59s Warning FailedCreate replicaset/nginx-1-658f4cf99f Error creating: pods "nginx-1-658f4cf99f-mccth" is forbidden: exceeded quota: pods-quota, requested: pods=1, used: pods=2, limited: pods=2 2m58s Warning FailedCreate replicaset/nginx-1-658f4cf99f Error creating: pods "nginx-1-658f4cf99f-z79vh" is forbidden: exceeded quota: pods-quota, requested: pods=1, used: pods=2, limited: pods=2 ...
The same can be checked using kubectl describe command where the used quota is same as hard limit so no more pods will be allowed to be created in this NS:
[root@controller ~]# kubectl describe ns pods-quota-ns
Name: pods-quota-ns
Labels: <none>
Annotations: <none>
Status: Active
Resource Quotas
Name: pods-quota
Resource Used Hard
-------- --- ---
pods 2 2
No LimitRange resource.
Understanding Limit Range
When you are using quotas on a namespace, one requirement is that every container in the namespace must have resource limits and requests defined. Sometimes this requirement can cause complexity and make it more difficult to work quickly with Kubernetes. Kubernetes provides the facility for default requests and limits to be provided at the namespace level. You could use this to provide some sensible defaults to namespaces used by a particular application or team.
We can configure default limits and requests for the containers in a namespace using the LimitRange object. This object allows us to provide defaults for the CPU or memory, or both. If a LimitRange object exists in a namespace, then any container created without the resource requests or limits configured in LimitRange will inherit these values from the limit range.
There are two situations where LimitRange will affect the resource limits or requests when a pod is created:
- Containers that have no resource limits or requests will inherit the resource limit and requests from the
LimitRangeobject - Containers that have no resource limits but do have requests specified will inherit the resource limit from the LimitRange object
If a container already has limits and requests defined, then LimitRange will have no effect. Because containers that specify only limits default the request field to the same value, they will not inherit the request value from LimitRange.
Example: Create and apply LimitRange to namespace
In this example we will create a LimitRange and assign it to our existing namespace pods-quota-ns where we had assigned a quota limit.
[root@controller ~]# cat assign-limit-range.yml apiVersion: v1 kind: LimitRange metadata: name: define-limit namespace: pods-quota-ns spec: limits: - default: memory: 512Mi cpu: 1 defaultRequest: memory: 256Mi cpu: 500m type: Container
Let's create this LimitRange:
[root@controller ~]# kubectl create -f assign-limit-range.yml
limitrange/define-limit created
To get the list of LimitRange in pods-quota-ns namespace:
[root@controller ~]# kubectl get limits -n pods-quota-ns NAME CREATED AT define-limit 2020-11-29T07:15:18Z
We can verify the LimitRange using kubectl describe:
[root@controller ~]# kubectl describe ns pods-quota-ns
Name: pods-quota-ns
Labels: <none>
Annotations: <none>
Status: Active
Resource Quotas
Name: pods-quota
Resource Used Hard
-------- --- ---
pods 2 2
Resource Limits
Type Resource Min Max Default Request Default Limit Max Limit/Request Ratio
---- -------- --- --- --------------- ------------- -----------------------
Container cpu - - 500m 1 -
Container memory - - 256Mi 512Mi -
Instead of using describe with namespace, we can use it with LimitRange directly:
[root@controller ~]# kubectl describe limits -n pods-quota-ns define-limit Name: define-limit Namespace: pods-quota-ns Type Resource Min Max Default Request Default Limit Max Limit/Request Ratio ---- -------- --- --- --------------- ------------- ----------------------- Container cpu - - 500m 1 - Container memory - - 256Mi 512Mi -
Now let's create a new deployment without specifying the resource limits (but before that I will delete the existing deployment which I created earlier):
[root@controller ~]# kubectl delete deployment -n pods-quota-ns nginx-1
deployment.apps "nginx-1" deleted
[root@controller ~]# kubectl get pods -n pods-quota-ns
NAME READY STATUS RESTARTS AGE
nginx-1 1/1 Running 0 25s
Now let's check the details of this pod to make sure if the default resource limit is applied:
[root@controller ~]# kubectl describe pods -n pods-quota-ns nginx-1
Name: nginx-1
Namespace: pods-quota-ns
Priority: 0
Node: worker-1.example.com/192.168.43.49
Start Time: Sun, 29 Nov 2020 13:03:22 +0530
Labels: run=nginx-1
...
State: Running
Started: Sun, 29 Nov 2020 13:03:34 +0530
Ready: True
Restart Count: 0
Limits:
cpu: 1
memory: 512Mi
Requests:
cpu: 500m
memory: 256Mi
...
As expected, even when we didn't provided any Resource Limits to this Pod, the pod has default LimitRange applied.
To delete the LimitRange we created in pods-quota-ns namespace:
[root@controller ~]# kubectl delete limitranges -n pods-quota-ns define-limit
limitrange "define-limit" deleted
Conclusion
In this Kubernetes tutorial we learned how we can define resource quota to namespace to handle resource scarcity across Cluseter nodes. If your namespace has a resource quota, it is helpful to have a default value in place for CPU limit. Here are two of the restrictions that a resource quota imposes on a namespace:
- Every Container that runs in the namespace must have its own CPU limit.
- The total amount of CPU used by all Containers in the namespace must not exceed a specified limit.
If a Container does not specify its own CPU limit, it is given the default limit, and then it can be allowed to run in a namespace that is restricted by a quota.
Also Read (External):
Configure Minimum and Maximum CPU Constraints for a Namespace
Configure Default CPU Requests and Limits for a Namespace
