Kubernetes Networking Tutorial [Beginner to PRO]

Networking plays a crucial role in a Kubernetes cluster as it enables seamless communication between applications running within the cluster. In a microservices architecture, applications are typically divided into smaller, independent services that work together to fulfill complex tasks. These services need to communicate with each other to exchange data, request resources, and coordinate their actions.

Kubernetes networking provides the necessary infrastructure to facilitate this communication. It ensures that each application or service within the cluster has a unique network identity and can discover and connect to other services as needed. Networking in Kubernetes enables both intra-cluster communication, allowing Pods to communicate with each other, and external communication, enabling access to services from outside the cluster.

By leveraging Kubernetes networking capabilities, applications can be horizontally scaled, allowing for the deployment of multiple instances of a service to handle increased traffic and ensure high availability. Networking also plays a vital role in load balancing, distributing incoming traffic across multiple service instances for efficient utilization of resources.

Furthermore, Kubernetes networking provides security mechanisms such as network policies, which enable fine-grained control over network traffic between services, ensuring that only authorized communication is allowed and protecting against potential threats.

At a high level, this tutorial covers the following concepts:

• Kubernetes networking basics
• Connectivity between Pods
• Services, service types and their endpoints
• Ingress controller and Ingress
• Using and configuring CoreDNS
• Choosing a container network interface (CNI) plugin

 

Kubernetes Networking Basics

The Kubernetes network model enables networking communication and needs to fulfill the following requirements:

1. Container-to-container communication: Containers running within the same Pod often need to communicate with each other. They can achieve this through various means, including Inter Process Communication (IPC) messages, file sharing, and direct communication via the loopback interface. By using the localhost hostname, containers within the same Pod can communicate with each other over the loopback network interface. Additionally, as each Pod is assigned a unique virtual IP address, containers within the same Pod share the same context and can communicate using that IP address and the shared port space.
2. Pod-to-Pod communication: Pods may need to communicate with other Pods within the same cluster, either on the same node or on different nodes. Kubernetes assigns a unique IP address to each Pod upon creation from the Pod CIDR (Classless Inter-Domain Routing) range of its respective node. It's important to note that these IP addresses are ephemeral and may change upon Pod restarts. Therefore, it is recommended to prefer Pod-to-Service communication over direct Pod-to-Pod communication for stability and reliability.
3. Pod-to-Service communication: Services in Kubernetes provide a single, stable DNS name for a group of Pods that serve a common purpose. Services act as an abstraction layer that enables load balancing requests across multiple Pods, providing high availability and scalability. Pod-to-Service communication allows Pods to access the Services by using the Service's DNS name. This communication can occur within the cluster or from outside the cluster, depending on the Service's configuration.
4. Node-to-node communication: Nodes within a Kubernetes cluster communicate with each other for various purposes, such as resource coordination and cluster management. Each node in the cluster is assigned a node IP address, which serves as its unique identifier. This allows nodes to establish communication and exchange information with other nodes within the cluster, facilitating tasks like workload distribution and coordination.

 

Connectivity between containers

• Containers within the same Pod share the same network namespace, enabling direct communication through the loopback interface.
• Each container within the Pod is allocated a unique IP address, allowing them to communicate with each other using this IP address and the shared port space.
• Containers can communicate via IPC (Inter Process Communication) mechanisms such as shared memory or signals.
• They can also share files and access shared volumes within the Pod.

Let's prove this, we will create a simple pod with 2 containers

apiVersion: v1
kind: Pod
metadata:
  name: sidecar-pod
spec:
  containers:
    - name: container1
      image: ubuntu
      command: ["/bin/sleep", "infinity"]
    - name: container2
      image: ubuntu
      command: ["/bin/sleep", "infinity"]

In this example, both container1 and container2 are using the ubuntu image. The command specified for each container is sleep infinity to keep them running indefinitely.

To connect to container1 from container2, you can use the localhost address (127.0.0.1) and the port on which container1 is listening. Here's an example:

We will use netcat to show the communication between containers. Start a shell session inside container1:

kubectl exec -it sidecar-pod -c container1 -- /bin/sh

To install the netcat-openbsd package inside the Pod, run the package installation command:

apt-get update
apt-get install -y netcat-openbsd

Connect to container1 using the localhost IP address and the port number on which container1 is listening (e.g., 8080):

nc -l -p 8080

Start a shell session inside container2:

kubectl exec -it sidecar-pod -c container2 -- /bin/bash

To install the netcat-openbsd package inside the Pod, run the package installation command:

apt-get update
apt-get install -y netcat-openbsd

Connect to the listening port in container1 (e.g., port 8080) from container2:

root@sidecar-pod:/# nc 127.0.0.1 8080 -vw2
Connection to 127.0.0.1 8080 port [tcp/*] succeeded!

As you can see we are able to communicate container1 from container2 using loopback address.

 

Connectivity between Pods

• Pods in Kubernetes are assigned unique IP addresses from the Pod CIDR range of their respective nodes upon creation.  You can inspect a Pod’s IP address by using the -o wide command-line option for the get pods command or by describing the Pod
• Each Pod can communicate with other Pods within the same cluster using their assigned IP addresses.
• Connectivity between Pods is established through the underlying network infrastructure provided by the container runtime and the networking plugin.
• Pods communicate with each other using their IP addresses without Network Address Translation (NAT) by default.
• Pod-to-Pod communication can occur regardless of the physical nodes on which the Pods are running.
• Kubernetes handles routing traffic between Pods by leveraging overlay networks or Software-Defined Networking (SDN) solutions.
• The Pod IP addresses are dynamic and can change during restarts or rescheduling, so it's important to rely on Service abstractions for stable network endpoints.
• Services act as a stable abstraction layer for a group of Pods, providing a single DNS name or IP address and load balancing capabilities.
• Pods can communicate with Services by using the Service's DNS name or IP address.
• Services enable load balancing and distribute incoming traffic across multiple Pods associated with the Service.
• Pod-to-Service communication can occur within the cluster or from outside the cluster, depending on the Service's configuration and exposure.
• Network Policies can be applied to regulate and control the flow of network traffic between Pods, allowing fine-grained access control.
• Network Policies define rules to permit or deny traffic based on source IP addresses, destination IP addresses, ports, and protocols.
• Network Policies help enforce security and isolation by restricting the communication between Pods based on defined rules.

Let me demonstrate this with an example.

I will create 2 pods and then will show you the connectivity between both the pods

Create a YAML file for Pod1 (pod1.yaml):

apiVersion: v1
kind: Pod
metadata:
  name: pod1
spec:
  containers:
    - name: container1
      image: nginx

Create a YAML file for Pod2 (pod2.yaml):

apiVersion: v1
kind: Pod
metadata:
  name: pod2
spec:
  containers:
    - name: container2
      image: nginx

Apply the YAML files to create the Pods:

kubectl apply -f pod1.yaml
kubectl apply -f pod2.yaml

Verify that the Pods are running and get their IP Address

$ kubectl get pods -o wide
NAME   READY   STATUS    RESTARTS   AGE   IP           NODE     NOMINATED NODE   READINESS GATES
pod1   1/1     Running   0          13s   10.244.1.2   node01   <none>           <none>
pod2   1/1     Running   0          13s   10.244.1.3   node01   <none>           <none>

Now verify the connectivity from pod1 to pod2 and vice versa:

$ kubectl exec -it pod1 -- curl 10.244.1.3
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>

--------------------

$ kubectl exec -it pod2 -- curl 10.244.1.2
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>

As you can see, we are able to access webserver deployed on pod2 via pod1 using the pod2 IP Address and vice versa.

But the important question is, should we use IP Address for pod-pod communication in cloud native world?

While it is possible to use IP addresses for communication between Pods in Kubernetes, it is generally not recommended to rely on IP addresses directly.

In Kubernetes, Pod IP addresses are subject to change, especially during Pod restarts or rescheduling. Relying on IP addresses for communication can lead to disruptions and unreliable connections.

Instead, Kubernetes provides the Service resource as an abstraction layer for Pods. Services offer stable endpoints with their own IP addresses or DNS names, simplifying service discovery and enabling load balancing. By utilizing Services, you can ensure consistent and reliable communication between Pods, even when the underlying Pod IP addresses change.

Let's understand more about Kubernetes Services.

 

How pod gets an IP Address?

Here's an outline of the process of how an IP Address gets assigned to a Pod automatically:

1. Container Runtime: Every time a pod is created, it's the job of the container runtime (Docker, containerd, etc.) to start the containers that make up the pod. Each container within a pod shares the network namespace, meaning they all use the same IP address and port space.
2. CNI Plugins: Kubernetes uses the Container Network Interface (CNI) as a standard for configuring networking interfaces for Linux containers. This is the layer at which the IP assignment happens. Depending on the cloud platform and container network interface (CNI) you use, this network might be an Amazon Web Services (AWS), Google Cloud Platform (GCP), or Azure virtual private cloud (VPC), or the basic flat network used by Kubernetes. In all cases, this principle is the same: each Pod is assigned its own IP address, and that is an absolute rule.
3. CNI Configuration and IPAM: Kubernetes delegates the job of assigning IP addresses to pods to the CNI plugin specified in the CNI configuration. Within the CNI, the IP address management (IPAM) is responsible for handling IP addressing for each pod. The exact way this happens depends on the CNI plugin and IPAM method you're using. Kubernetes does not come with a default CNI plugin; popular options include Calico, Weave, Cilium, and others.
4. IP Address Assignment: When a pod is created, the kubelet on the node (the agent that communicates with the master node) sends a request to the configured CNI plugin to set up a network for the pod. The CNI plugin, in turn, asks the IPAM for an available IP address. Once an IP address is assigned, the CNI plugin sets up the network namespace for the pod, applies the IP address and routes to it, and stores the details for cleanup when the pod is removed. IP addresses may change when Pods are restarted, rescheduled, or during scaling operations.

To check the Pod CIDR range defined in Kubernetes and view the available IP pool for Pod IP address allocation, you can use the following command:

kubectl cluster-info dump | grep cluster-cidr

This command retrieves the cluster information dump and filters for the line that specifies the cluster CIDR range. The cluster CIDR range represents the IP address range from which the networking plugin allocates IP addresses to Pods.

Sample Output

 "--cluster-cidr=10.244.0.0/16",

 

 

Understanding Kubernetes Services

Kubernetes Services play a crucial role in providing stable network endpoints for Pods within a cluster. Services act as an abstraction layer that decouples the underlying Pods from the consumers, enabling seamless communication and load balancing.

The below illustrates the functionality. Pod 1 receives traffic as its assigned labels match with the label selection defined in the Service. Pod 2 does not receive traffic as it defines nonmatching labels. Note that it is possible to create a Service without a label selector for less-common scenarios.

 

Let's dive deeper into the concept of Kubernetes Services and the different types available:

When to use each type of Service:

• ClusterIP: Use ClusterIP Services for internal communication between Pods or Services within the cluster. They are suitable for internal microservices communication and should be used when external access is not required.
• NodePort: NodePort Services are useful when external access is needed, such as for development or testing purposes. However, they may not be recommended for production environments due to potential security risks and port conflicts.
• LoadBalancer: LoadBalancer Services are ideal for production environments that require external access and scalability. They provide an external IP address and automatic load balancing, making them suitable for high-traffic applications.

 

Working with Kubernetes Services

Creating Helm Deployment

Let's get our hands dirty and start working with Kubernetes services with a hands on. We will use HELM Charts to create a deployment, service and then accessing the service.

First, you create a new chart:

helm create mychart

You'll find several files in the mychart/templates directory. You need to update mychart/templates/deployment.yaml.

Update the deployment.yaml file with following content:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ include "mychart.fullname" . }}
  labels:
    {{- include "mychart.labels" . | nindent 4 }}
spec:
  replicas: {{ .Values.replicaCount }}
  selector:
    matchLabels:
      {{- include "mychart.selectorLabels" . | nindent 6 }}
  template:
    metadata:
      labels:
        {{- include "mychart.selectorLabels" . | nindent 8 }}
    spec:
      containers:
        - name: {{ .Chart.Name }}
          image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
          ports:
            - name: http
              containerPort: 80
              protocol: TCP

You should also update mychart/templates/service.yaml.

apiVersion: v1
kind: Service
metadata:
  name: {{ include "mychart.fullname" . }}
  labels:
    {{- include "mychart.labels" . | nindent 4 }}
spec:
  type: ClusterIP
  ports:
    - port: 80
      targetPort: http
      protocol: TCP
      name: http
  selector:
    {{- include "mychart.selectorLabels" . | nindent 4 }}

Deploy your Helm chart:

helm install my-release ./mychart

NAME: my-release
LAST DEPLOYED: Thu Jun 29 08:58:05 2023
NAMESPACE: default
STATUS: deployed
REVISION: 1
NOTES:
1. Get the application URL by running these commands:
  export POD_NAME=$(kubectl get pods --namespace default -l "app.kubernetes.io/name=mychart,app.kubernetes.io/instance=my-release" -o jsonpath="{.items[0].metadata.name}")
  export CONTAINER_PORT=$(kubectl get pod --namespace default $POD_NAME -o jsonpath="{.spec.containers[0].ports[0].containerPort}")
  echo "Visit http://127.0.0.1:8080 to use your application"
  kubectl --namespace default port-forward $POD_NAME 8080:$CONTAINER_PORT

 

Listing all Kubernetes Services

You can list all services within the current namespace by executing:

kubectl get services

Listing all Services presents a table view that includes the Service type, the cluster IP address, and the incoming port. Here, you can see the output for the my-release chart we created earlier:

NAME                 TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
kubernetes           ClusterIP   10.96.0.1      <none>        443/TCP   17m
my-release-mychart   ClusterIP   10.105.24.11   <none>        80/TCP    24s

 

Rendering Kubernetes Service

You may want to drill into the details of a Service for troubleshooting purposes. That might be the case if the incoming traffic to a Service isn’t routed properly to the set of Pods you expect to handle the workload.

The describe command renders valuable information about the configuration of a Service. The configuration relevant to troubleshooting a Service is the value of the fields Selector, IP, Port, TargetPort, and Endpoints.

kubectl describe svc my-release-mychart

Name:              my-release-mychart
Namespace:         default
Labels:            app.kubernetes.io/instance=my-release
                   app.kubernetes.io/managed-by=Helm
                   app.kubernetes.io/name=mychart
                   app.kubernetes.io/version=1.16.0
                   helm.sh/chart=mychart-0.1.0
Annotations:       meta.helm.sh/release-name: my-release
                   meta.helm.sh/release-namespace: default
Selector:          app.kubernetes.io/instance=my-release,app.kubernetes.io/name=mychart
Type:              ClusterIP
IP Family Policy:  SingleStack
IP Families:       IPv4
IP:                10.105.24.11
IPs:               10.105.24.11
Port:              http  80/TCP
TargetPort:        http/TCP
Endpoints:         10.244.1.3:80
Session Affinity:  None
Events:            <none>

 

Exploring Endpoints

In Kubernetes, an Endpoint object represents a group of IP addresses and ports which can be used to access a service. It is automatically managed by Kubernetes when a service is created, updated, or deleted.

The Endpoint is responsible for routing traffic from the service to the appropriate pods. It contains the IP addresses of the pods that the service should send traffic to. If you have multiple pods that are part of the service, then each of those pod's IP addresses would be part of the Endpoint object.

The following command lists the endpoint for the Service named my-release-mychart:

NAME                 ENDPOINTS          AGE
kubernetes           172.20.68.5:6443   24m
my-release-mychart   10.244.1.3:80      7m23s

The details of the endpoint give away the full list of IP address and port combinations

$ kubectl describe endpoints my-release-mychart
Name:         my-release-mychart
Namespace:    default
Labels:       app.kubernetes.io/instance=my-release
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=mychart
              app.kubernetes.io/version=1.16.0
              helm.sh/chart=mychart-0.1.0
Annotations:  endpoints.kubernetes.io/last-change-trigger-time: 2023-06-29T08:58:06Z
Subsets:
  Addresses:          10.244.1.3
  NotReadyAddresses:  <none>
  Ports:
    Name  Port  Protocol
    ----  ----  --------
    http  80    TCP

Events:  <none>

Here the Addresses contains the IP Address of the Pod so the service will redirect the traffic to the Pod associated with Address 10.244.1.3

$ kubectl get po -o wide
NAME                                  READY   STATUS    RESTARTS   AGE     IP           NODE     NOMINATED NODE   READINESS GATES
my-release-mychart-7d5ff6c587-lpq6l   1/1     Running   0          9m49s   10.244.1.3   node01   <none>           <none>

 

Port Mapping

In Kubernetes Networking, port mapping is a critical aspect of how network traffic is directed from services to pods. When you create a service in Kubernetes, you typically specify two kinds of ports:

1. Port: This is the port on which the service is exposed and can receive traffic. This port is accessible within the cluster and, depending on the service type, from outside the cluster as well.
2. TargetPort: This is the port on the pod where the network traffic will be received. This port needs to match a container port in the pod that the service should send traffic to.

When a service is created in Kubernetes, an Endpoint object is automatically created as well. The Endpoint contains the IP addresses of the pods that the service should send traffic to. When a network request is made to the service on the Port, Kubernetes will forward that request to the TargetPort on one of the pod IP addresses listed in the Endpoint.

For example, if a service has Port 80 and TargetPort 8080, any network traffic that comes into the service on port 80 will be routed to port 8080 on one of the pods that the service is responsible for.

 

Accessing a Service with Type ClusterIP

A ClusterIP service in Kubernetes Networking is the default Kubernetes service. It provides a service inside your cluster that other apps can access. ClusterIP exposes the service on a cluster-internal IP, making it only reachable from within the cluster.

Following image illustrates the accessibility of a Service with type ClusterIP

We will create a Pod and a corresponding Service to demonstrate the runtime behavior of ClusterIP

Let's create a simple deployment with a single replica running the nginx image. This will serve as the backend for our service.

Save the following to nginx-deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80

Apply the deployment with kubectl apply -f nginx-deployment.yaml.

deployment.apps/nginx-deployment created

Now let's create a service that targets the nginx deployment. The selector should match the labels of our deployment's pods. Save the following to nginx-service.yaml:

apiVersion: v1
kind: Service
metadata:
  name: nginx-service
spec:
  selector:
    app: nginx
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80

Apply the service with kubectl apply -f nginx-service.yaml.

service/nginx-service created

Next, let's create a simple pod to access the service. This pod will use the curlimages/curl image, which has the curl tool available.

Save the following to curl-pod.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: curl-pod
spec:
  containers:
  - name: curl-container
    image: curlimages/curl
    command: ['sh', '-c', 'sleep 3600']

Apply the pod with kubectl apply -f curl-pod.yaml.

pod/curl-pod created

After the curl-pod is running, you can use kubectl exec to run a curl command from inside the curl-pod:

kubectl exec curl-pod -- curl http://nginx-service

This command sends a request to the nginx-service. Since services are available across the cluster via DNS, you can just use the service name as the hostname. The curl command should return the default nginx welcome page, demonstrating that the nginx-service is accessible from the curl-pod.

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   612  100   612    0     0  54308      <!DOCTYPE html>-:-- --:--:--     0
<html>
<head>
<title>Welcome to nginx!</title>

A Kubernetes service of type ClusterIP exposes the service on a cluster-internal IP, and it's only reachable from within the cluster. This means you can access it from a different pod inside the cluster, but not from outside the cluster.

If you need to expose a service to external traffic, you should use a service of type NodePort, LoadBalancer, or Ingress, depending on your specific requirements and the environment in which your cluster is running.

 

Accessing a Service with Type NodePort

A Kubernetes service of type NodePort exposes the service on a port on each node of the cluster. This service is accessible both from within the cluster and from outside the cluster.

• When a service of type NodePort is created, Kubernetes allocates a port from a default range (30000-32767) on each node for that service. You can also specify a particular port from this range.
• Each node will proxy that allocated port, regardless of whether there are pods from the service running on the node.
• The service is accessible through the IP address of the node as well as the allocated port. In other words, if you have a service of type NodePort running on your cluster and you can reach any node of the cluster using its IP address, you can access the service.
• Inside the service specification, the nodePort field corresponds to the port on each node, the port field is the port on the service, and targetPort is the port on the pod where the network traffic will be received.

Let's create a NodePort service that exposes the nginx-deployment we created previously.

You can use the following YAML to create a NodePort service. Save the following to nginx-nodeport-service.yaml:

apiVersion: v1
kind: Service
metadata:
  name: nginx-nodeport-service
spec:
  type: NodePort
  selector:
    app: nginx
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
      nodePort: 30080

In this YAML, nodePort is the port on which the service will be available on each node in your cluster. It must be a valid port number in the range of 30000-32767.

You can create the service using kubectl apply -f nginx-nodeport-service.yaml.

service/nginx-nodeport-service created

Describe and get the service details:

$ kubectl describe svc nginx-nodeport-service
Name:                     nginx-nodeport-service
Namespace:                default
Labels:                   <none>
Annotations:              <none>
Selector:                 app=nginx
Type:                     NodePort
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       10.111.193.174
IPs:                      10.111.193.174
Port:                     <unset>  80/TCP
TargetPort:               80/TCP
NodePort:                 <unset>  30080/TCP
Endpoints:                10.244.1.4:80
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>

Access the Service from a Pod

Like before, you can use the curl-pod to access the service from within the cluster:

kubectl exec curl-pod -- curl http://nginx-nodeport-service

This should return the default nginx welcome page.

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>

Access the Service from Outside the Cluster

To access the service from outside the cluster, you need to know the IP address of one of your nodes and the nodePort. You can get your nodes' IP addresses using kubectl get nodes -o wide.

Once you have a node IP and the nodePort (which is 30080 as per our service definition), you can access the service using <node-ip>:<node-port>.

Here our deployment pod is deployed on node01

$ kubectl get po -o wide
NAME                                READY   STATUS    RESTARTS   AGE     IP           NODE     NOMINATED NODE   READINESS GATES
curl-pod                            1/1     Running   0          71s     10.244.1.5   node01   <none>           <none>
nginx-deployment-85996f8dbd-sgk2s   1/1     Running   0          2m19s   10.244.1.4   node01   <none>           <none>


root@controlplane:~$ kubectl get nodes -o wide
NAME           STATUS   ROLES           AGE   VERSION   INTERNAL-IP    EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
controlplane   Ready    control-plane   15m   v1.26.0   172.20.128.5   <none>        Ubuntu 22.04.2 LTS   5.15.0-72-generic   containerd://1.6.12
node01         Ready    <none>          14m   v1.26.0   172.20.128.6   <none>        Ubuntu 22.04.2 LTS   5.15.0-72-generic   containerd://1.6.12

In our case the node01 IP is 172.20.128.6, so we can access the service by using 172.20.128.6:30080:

$ curl http://172.20.128.6:30080
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>

As expected we are able to access our service using kubernetes node from outside the cluster.

 

Accessing a Service with Type LoadBalancer

A Kubernetes Service of type LoadBalancer is an extension of the NodePort service type and it is used to expose your service to the internet. When you create a LoadBalancer service, Kubernetes provisions a cloud network load balancer for you, and creates a stable, reliable IP address that sends traffic to the NodePort and ClusterIP services created by Kubernetes.

 

Here's an example of a LoadBalancer service:

apiVersion: v1
kind: Service
metadata:
  name: nginx-loadbalancer-service
spec:
  type: LoadBalancer
  selector:
    app: nginx
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80

In this service definition:

• spec.type is set to LoadBalancer, which means a cloud load balancer will be provisioned for this service (if supported by the underlying infrastructure).
• spec.selector is set to match the labels of the nginx pods. This means the service will route traffic to the nginx pods.
• spec.ports defines the ports for the service. port is the port that the service will listen on, and targetPort is the port on the pods that the service will forward traffic to.

To create this service, save the YAML to a file called nginx-loadbalancer-service.yaml and run:

service/my-loadbalancer-service created

To check the status of the service and get the external IP, use:

$ kubectl get service my-loadbalancer-service
NAME                      TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
my-loadbalancer-service   LoadBalancer   10.102.236.58   10.109.76.157     80:31706/TCP   32s

If the EXTERNAL-IP status is marked as <pending> then that implies that the cloud provider's load balancer has not yet provisioned an IP address for your service. This might be because your Kubernetes cluster is not running in an environment that supports LoadBalancer services.

If your cluster is running in a cloud provider that supports automatic provisioning of a load balancer (like AWS, Google Cloud, Azure, etc.), the <pending> status usually changes to an actual public IP address after some time. If your cluster is not running in such an environment (like a local Minikube setup or some other configurations), the external IP address will never be provisioned, and the status will remain <pending>.

To call the Service from outside of the cluster, use the external IP address and its incoming port. In our case as we have specified exposed port as 80 so we need not specify port number explicitly:

curl http://<external-ip>

 

Accessing a Service with Type Headless

In Kubernetes Networking, a Service is an abstraction which defines a logical set of Pods and a policy by which to access them - sometimes called a micro-service. The set of Pods targeted by a Service is usually determined by a selector.

Services without selectors and those with a None cluster IP are called headless. This means they don't have a cluster IP and cannot be accessed through the standard round-robin load balancing like standard Services. DNS is configured to return multiple A records (addresses), one for each Pod in the headless Service.

Headless services are useful when you want to:

1. Reach every Pod directly for systems that require node-to-node communication, or for StatefulSets.
2. Use other types of service discovery mechanisms, without wanting to deal with the default round-robin TCP proxying of Kubernetes Services.

Here's how you would define a headless service:

apiVersion: v1
kind: Service
metadata:
  name: nginx-headless
  labels:
    app: nginx
spec:
  ports:
  - port: 80
  clusterIP: None
  selector:
    app: nginx

Here clusterIP: None is what makes the service headless.

Now let's assume you've created a StatefulSet with Pods labelled with app: nginx. These pods will be exposed through the headless service:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: web
spec:
  serviceName: "nginx-headless"
  replicas: 3
  selector:
    matchLabels:
      app: nginx 
  template:
    metadata:
      labels:
        app: nginx 
    spec:
      containers:
      - name: nginx
        image: nginx
        ports:
        - containerPort: 80
          name: web

The pods will be reachable by the name <statefulset-name>-<ordinal>.<service-name>.<namespace>.svc.cluster.local. In this case, the pods will be named web-0, web-1, and web-2 respectively.

Let's create these resources:

statefulset.apps/web created
service/nginx-headless created

Verify the resources:

root@controlplane:~$ kubectl get svc  nginx-headless
NAME             TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
nginx-headless   ClusterIP   None         <none>        80/TCP    38s


root@controlplane:~$ kubectl get po
NAME    READY   STATUS    RESTARTS   AGE
web-0   1/1     Running   0          50s
web-1   1/1     Running   0          43s
web-2   1/1     Running   0          41s

I have another pod (curl-pod) which I will use to query the headless nginx service:

kubectl exec curl-pod -- curl web-0.nginx-headless.default.svc.cluster.local
kubectl exec curl-pod -- curl web-1.nginx-headless.default.svc.cluster.local
kubectl exec curl-pod -- curl web-2.nginx-headless.default.svc.cluster.local

Sample Output:

$ kubectl exec curl-pod -- curl web-0.nginx-headless.default.svc.cluster.local
  % Total    % Received % Xfer<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>

---------
$ kubectl exec curl-pod -- curl web-1.nginx-headless.default.svc.cluster.local
  % Total    % Received % Xfer<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>

----------

$ kubectl exec curl-pod -- curl web-2.nginx-headless.default.svc.cluster.local
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>

 

Understanding Kubernetes Ingress

In Kubernetes Networking, an Ingress is an API object that acts as a gateway or entry point for external traffic into a cluster. It provides a way to manage external access to services running within the cluster. In other words, Ingress allows you to expose your services to the outside world and route incoming traffic to the appropriate backend services based on specific rules.

The role of Ingress in Kubernetes is to abstract the complexity of managing individual service exposure and load balancing. It provides a centralized and configurable way to control the traffic flow into the cluster, making it easier to manage external access and routing.

I have covered Kubernetes Ingress in detail with examples in a different article Steps to expose services using Kubernetes Ingress

 

Understanding Kubernetes CoreDNS

CoreDNS is a fast and flexible DNS server used in Kubernetes Networking. It's written in Go and built around a plug-in-based architecture, which makes it easily extendable. Each functionality of CoreDNS is provided by a variety of plugins that can be chained in a pipeline to serve DNS requests. This design allows users to customize the CoreDNS server to fit specific use cases.

 

CoreDNS Architecture

The architecture of CoreDNS is quite simple and elegant, revolving around the concept of middleware or, as they are now referred to, plugins.

• Plugins: The functionality of CoreDNS is extended through plugins. Each plugin performs a DNS function, such as Kubernetes service discovery, Prometheus metrics, caching, rewriting queries, and more. They can be chained together in any order in the Corefile (the configuration file for CoreDNS), which is read at startup. The order in which these plugins are chained is important, as each plugin may interact with the DNS request and response.
• Corefile: This is the configuration file for CoreDNS, written in a syntax similar to a typical INI file or an Apache .htaccess file. The Corefile specifies which plugins are active and in which order they should process requests. The global parameters like the server's port are also defined in the Corefile.
• Server Blocks: CoreDNS uses server blocks in the Corefile to define different servers that might have different zones they are authoritative for, and different sets of middleware.

Here's a simple example of how CoreDNS architecture works:

Let's say you have this simple Corefile:

.:53 {
    errors
    health
    kubernetes cluster.local in-addr.arpa ip6.arpa
    prometheus
    forward . /etc/resolv.conf
    cache 30
    loop
    reload
    loadbalance
}

In this Corefile, we are defining one server block (listening on port 53). It has several plugins enabled:

• errors: This will return error messages in the response.
• health: This enables a health check HTTP endpoint on port 8080.
• kubernetes: This enables CoreDNS to reply to DNS queries based on IP of the services in Kubernetes.
• prometheus: This enables metrics gathering to be readable by Prometheus.
• forward: This forwards queries to a set of predefined upstream resolvers (in this case, the ones defined in /etc/resolv.conf).
• cache 30: This enables a cache that caches responses for 30 seconds.
• loop: This checks for loops in DNS resolution.
• reload: This allows automatic reload of a changed Corefile.
• loadbalance: This is for round robin load balancing of A and AAAA records.

When a DNS request comes in, CoreDNS will pass the request to the first plugin (in this case, the errors plugin), then to the next one (the health plugin), and so on down the chain.

 

Installation and Configuration

Since CoreDNS is the default DNS service in Kubernetes, it comes pre-installed with the cluster. If you set up a Kubernetes cluster using kubeadm or similar tools, CoreDNS is deployed automatically as a part of the cluster. If you need to install it manually or want to replace kube-dns with CoreDNS, you can use the CoreDNS deployment yaml files provided in the official CoreDNS GitHub repository.

CoreDNS in a Kubernetes cluster is configured through a ConfigMap resource. This allows the CoreDNS configuration to be updated dynamically without needing to rebuild or redeploy the CoreDNS image.

The ConfigMap is typically named coredns and resides in the kube-system namespace.

Here's how to view the CoreDNS ConfigMap:

$ kubectl -n kube-system get configmap coredns -o yaml
apiVersion: v1
data:
  Corefile: |
    .:53 {
        errors
        health {
           lameduck 5s
        }
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
           pods insecure
           fallthrough in-addr.arpa ip6.arpa
           ttl 30
        }
        prometheus :9153
        forward . /etc/resolv.conf {
           max_concurrent 1000
        }
        cache 30
        loop
        reload
        loadbalance
    }
kind: ConfigMap
metadata:
  creationTimestamp: "2023-06-29T10:47:18Z"
  name: coredns
  namespace: kube-system
  resourceVersion: "238"
  uid: 59b06a63-6e9b-4e4c-953d-342339c1e6cc

The ConfigMap will typically contain a Corefile section, which defines the behavior of the CoreDNS server. The Corefile is made up of multiple "plugins", and the sequence of plugins matters.

In the above configuration, each keyword (e.g., errors, health, kubernetes, prometheus, forward, cache, loop, reload, loadbalance) corresponds to a plugin, which is used to extend CoreDNS's functionality.

 

CoreDNS Plugins

CoreDNS has a flexible architecture, and its functionality is extended through plugins. These plugins can perform various functions related to DNS, such as modifying requests or responses, providing health checks, and more. Let's deep dive into some of these useful plugins:

• prometheus: This plugin enables metrics reporting in the Prometheus format. The exposed metrics can be scraped by a Prometheus server and visualized with Grafana or other Prometheus-compatible systems. Metrics like request count, request duration, and response size are collected.
• kubernetes: The kubernetes plugin enables CoreDNS to serve DNS records for Kubernetes services. This allows applications in the cluster to discover each other by DNS name. The plugin can also serve DNS records for the Pod IPs. It is an essential plugin for Kubernetes as it provides service discovery within the cluster.
• health: The health plugin provides a simple HTTP health check endpoint on a specified port, often port 8080. When accessed, it returns a 200 OK HTTP status, confirming that the CoreDNS server is up and running.
• cache: The cache plugin adds caching functionality, storing DNS responses for a specified time. This can improve performance, as subsequent requests for the same domain won't need to be forwarded to the upstream DNS server but can be served from the cache.
• rewrite: The rewrite plugin allows the modification of requests and responses. This can be used to change the domain name in requests, modify response records, change the query type, etc. For example, you can configure CoreDNS to rewrite requests for "service.namespace" to "service.namespace.svc.cluster.local", which can be useful in certain scenarios.
• forward: The forward plugin is used to forward DNS queries to an upstream resolver. It’s used when a query is outside of the cluster domain, or when a specific zone is defined to be handled by a different DNS server. It's comparable to the proxy functionality in the previous kube-dns.
• loop: This plugin checks for loops in DNS resolution paths. It sends a "random" A query to itself and verifies if the query comes back. If it sees the same query twice, then there is a loop and CoreDNS will stop to prevent a DNS forwarding loop.
• errors: This plugin ensures that all errors reaching this plugin are logged. This helps to identify and debug issues with DNS resolution.

 

Customizing CoreDNS

Customizing CoreDNS in Kubernetes Networking involves editing its ConfigMap. The ConfigMap is usually named coredns and resides in the kube-system namespace. You can edit the ConfigMap using the command:

kubectl -n kube-system edit configmap coredns

This will open the ConfigMap in your default text editor. The changes you make to the ConfigMap will take effect after you save and exit the editor.

1. Adding Custom DNS Entries

To add custom DNS entries, we will use the hosts plugin. Suppose we want to resolve myapp.local to 192.168.0.123. We will add the hosts plugin to the Corefile like this:

apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns
  namespace: kube-system
data:
  Corefile: |
    .:53 {
        errors
        health
        kubernetes cluster.local in-addr.arpa ip6.arpa {
           pods insecure
           fallthrough in-addr.arpa ip6.arpa
        }
        hosts {
            192.168.0.123 myapp.local
            fallthrough
        }
        prometheus :9153
        forward . /etc/resolv.conf
        cache 30
        loop
        reload
        loadbalance
    }

 

2. Forwarding DNS Queries

The forward plugin can be used to forward DNS queries for certain domains to specific DNS servers. Suppose you want to forward all DNS queries for mydomain.com to a DNS server running at 10.0.0.1. Here is how you would modify the Corefile:

apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns
  namespace: kube-system
data:
  Corefile: |
    .:53 {
        errors
        health
        kubernetes cluster.local in-addr.arpa ip6.arpa {
           pods insecure
           fallthrough in-addr.arpa ip6.arpa
        }
        prometheus :9153
        cache 30
        loop
        reload
        loadbalance
    }
    mydomain.com:53 {
        errors
        forward . 10.0.0.1
        cache 30
    }

In the above configuration, the first part is the default configuration for the cluster.local domain, and the second part is our added configuration for mydomain.com.

After editing the ConfigMap, CoreDNS pods need to be restarted to pick up the changes. If you are in a non-production environment, you can simply delete the CoreDNS pods, and they will be recreated automatically with the new configuration:

kubectl -n kube-system delete pod -l k8s-app=kube-dns

 

Comparison with kube-dns

• Architecture: CoreDNS has a more flexible and extensible architecture, based on a plugin mechanism. This allows it to support a wide range of DNS record types and features. Kube-DNS, on the other hand, is based on a fixed, monolithic design that lacks this flexibility.
• Performance: CoreDNS generally offers better performance and lower memory usage than Kube-DNS. It is written in Go and designed to be lightweight and efficient.
• DNS Record Support: CoreDNS supports a wider variety of DNS record types. Besides the standard A and PTR records, it can also serve SRV, TXT, and MX records, among others. Kube-DNS has more limited support for DNS record types.
• Service Discovery: Both CoreDNS and Kube-DNS provide service discovery features in a Kubernetes environment, but CoreDNS does so in a more flexible and configurable way due to its plugin architecture.
• DNS Forwarding and Fallback: CoreDNS supports flexible DNS forwarding and can use a different resolver for each DNS zone. It also has a robust fallback mechanism. Kube-DNS supports forwarding but is less flexible.

 

Choosing an Appropriate Container Network Interface Plugin

When setting up a Kubernetes cluster, one of the essential components to decide upon is the Container Network Interface (CNI) plugin to use. CNI is a standard that allows various networking technologies to plug into Kubernetes and other container orchestration systems.

The chosen CNI plugin influences the networking capabilities of your Kubernetes cluster, affecting network performance, functionality, and security. This guide provides an overview of factors to consider when choosing a CNI plugin for your Kubernetes setup.

• Networking Model: Understand whether your Kubernetes setup requires a flat or overlay networking model. A flat network typically offers higher speed, while an overlay network is easier to manage in large clusters.
• Network Policy Support: If your setup requires enforcing network policies, make sure the CNI plugin supports the Kubernetes NetworkPolicy API.
• Performance: The CNI plugin can greatly impact network latency and bandwidth. For instance, plugins using overlay networks may add latency, while others like Calico avoid this overhead and provide better performance.
• Scalability: Ensure your CNI plugin can scale as per your application needs. Some plugins, like Calico that uses BGP for networking, handle scaling more effectively.
• Community Support: Choosing a plugin with a vibrant community and robust ecosystem ensures ongoing support and frequent updates.
• Integration: If your cluster needs to work with existing infrastructure like firewalls or load balancers, ensure the chosen CNI plugin is compatible.
• Ease of Use: Consider the ease of installation and management. Some plugins like Flannel are simple to install, while others like Calico or Cilium may offer more features but could be complex to manage.

Choosing a CNI plugin depends on your specific requirements. Popular options include Calico, Cilium, Flannel, Weave, and Kube-router

 

Summary

This guide covers the fundamentals of Kubernetes networking, from basic pod and container connectivity to advanced service types and DNS configuration. We discussed how IP addresses are assigned to pods and explored various service types, including ClusterIP, NodePort, LoadBalancer, and Headless services. We also explained Kubernetes Ingress and dove deep into the workings of CoreDNS, discussing its architecture, installation, configuration, customization, and comparison with kube-dns. Lastly, we offered insights into choosing the appropriate Container Network Interface (CNI) Plugin based on various requirements.

You can read more at the official page of Kubernetes Networking

 

Key Takeaways

• Kubernetes ensures networking connectivity between containers and pods, and assigns unique IP addresses to each pod.
• Kubernetes Services provide a stable, abstracted interface to a set of pods, with various access modes defined by service types (ClusterIP, NodePort, LoadBalancer, and Headless).
• Ingress in Kubernetes is a way to manage access to services in a cluster from outside.
• CoreDNS is a flexible, extensible DNS server that can be configured to serve custom DNS entries, forward DNS queries to other servers, and serve as a Kubernetes cluster DNS.
• When choosing a CNI plugin, consider factors like networking model, network policy support, performance, scalability, community support, infrastructure integration, and ease of use.