Troubleshoot a Kubernetes Service That Is Not Working

You can see the Service in kubectl get and still get connection refused, timeouts, or empty replies. Work from the application Pod outward: confirm the process listens on the right port, test Pod IP directly, verify EndpointSlices, then ClusterIP, DNS, and external exposure.

Application process → Pod IP → EndpointSlice → ClusterIP → DNS → external exposure

Quick reference

Test from a client Pod inside the cluster before you curl NodePort or LoadBalancer from your workstation. Replace NS, SVC, POD, and PORT with your values.

Prepare a Reproducible Service Lab

Deploy the backend and Service

The walkthrough uses a two-replica nginx web Deployment and ClusterIP Service in svc-trouble-lab. Apply this manifest before the client Pod:

apiVersion: v1
kind: Namespace
metadata:
  name: svc-trouble-lab
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
  namespace: svc-trouble-lab
spec:
  replicas: 2
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
      - name: app
        image: nginx:1.27-alpine
        command: ["/bin/sh", "-c"]
        args:
        - |
          echo "server $(hostname)" > /usr/share/nginx/html/index.html
          touch /tmp/healthy
          exec nginx -g "daemon off;"
        ports:
        - name: http
          containerPort: 80
        readinessProbe:
          exec:
            command: ["/bin/sh", "-c", "test -f /tmp/healthy"]
          initialDelaySeconds: 2
          periodSeconds: 3
---
apiVersion: v1
kind: Service
metadata:
  name: web
  namespace: svc-trouble-lab
spec:
  selector:
    app: web
  ports:
  - name: http
    port: 80
    targetPort: http

kubectl apply -f web-lab.yaml

kubectl wait -n svc-trouble-lab --for=condition=Available deployment/web --timeout=180s

For Service types and EndpointSlice basics, see Kubernetes Services.

Create the client Pod

Testing only from your workstation cannot separate ClusterIP routing problems from NodePort, LoadBalancer, or Ingress issues. Run network checks from a Pod inside the cluster first. For how Services, CNI, and kube-proxy relate before you test routing, see Kubernetes networking.

This workflow uses a lightweight client image with curl. The curl command covers -w, --max-time, and exit codes for quick HTTP checks from Pods. Use the dedicated DNS troubleshooting guide when you need nslookup or dig. Add this Pod to the same namespace:

apiVersion: v1
kind: Pod
metadata:
  name: net-client
  namespace: svc-trouble-lab
spec:
  containers:
  - name: tools
    image: curlimages/curl:8.5.0
    command: ["sleep", "infinity"]

Apply the client Pod:

kubectl apply -f net-client.yaml

kubectl wait -n svc-trouble-lab --for=condition=Ready pod/net-client --timeout=120s

Keep net-client running so every test uses the same network path.

Capture Service, Pod, and EndpointSlice values

Before you change anything, capture the current Service, Pod, and EndpointSlice state.

List the Service with type and ClusterIP:

kubectl get service web -n svc-trouble-lab -o wide

Read ports, selector, and legacy Endpoints summary:

kubectl describe service web -n svc-trouble-lab

List backend Pods with labels and IPs:

kubectl get pods -n svc-trouble-lab -o wide --show-labels

Inspect EndpointSlices for the Service name:

kubectl get endpointslice -n svc-trouble-lab -l kubernetes.io/service-name=web

After the Deployment is ready, capture live values for later tests:

SERVICE_IP=$(kubectl get service web -n svc-trouble-lab -o jsonpath='{.spec.clusterIP}')

BACKEND_POD=$(kubectl get pods -n svc-trouble-lab -l app=web -o jsonpath='{.items[0].metadata.name}')

BACKEND_IP=$(kubectl get pod "$BACKEND_POD" -n svc-trouble-lab -o jsonpath='{.status.podIP}')

Record namespace, SERVICE_IP, Service port, target port, protocol, selector, Pod IPs, readiness, and EndpointSlice addresses. You will compare later steps against this baseline.

Test ClusterIP and Service DNS

From net-client, test the short DNS name and the ClusterIP on the Service port:

kubectl exec -n svc-trouble-lab net-client -- curl -s -o /dev/null -w "%{http_code}\n" "http://web:80"

Sample output:

200

Test the ClusterIP directly:

kubectl exec -n svc-trouble-lab net-client -- curl -s -o /dev/null -w "%{http_code}\n" "http://${SERVICE_IP}:80"

Sample output:

200

Test the fully qualified Service name. Kubernetes defines the full name as <service>.<namespace>.svc.<cluster-domain>. This kubeadm lab uses the default cluster.local domain. On clusters with a custom domain, read the search entries in the client Pod's /etc/resolv.conf and substitute that domain.

kubectl exec -n svc-trouble-lab net-client -- cat /etc/resolv.conf

kubectl exec \
  -n svc-trouble-lab net-client \
  -- curl \
  -s \
  -o /dev/null \
  -w "%{http_code}\n" "http://web.svc-trouble-lab.svc.cluster.local."

Sample output:

200

Kubernetes Service records use the configured cluster domain rather than requiring cluster.local.

Interpret the pair:

HTTP 200 on both short name and ClusterIP confirms in-cluster Service routing before you debug external exposure.

Diagnose EndpointSlice Backends

No addresses versus no ready endpoints

EndpointSlice addresses and readiness are separate checks. No addresses usually means the selector matches no Pods. Addresses with ready=false mean the Pods were selected but are not currently eligible for normal Service traffic.

kubectl get endpointslice -n svc-trouble-lab -l kubernetes.io/service-name=web -o wide

Inspect each address and its ready condition:

kubectl get endpointslice \
  -n svc-trouble-lab \
  -l kubernetes.io/service-name=web \
  -o jsonpath='{range .items[*].endpoints[*]}{.addresses[0]}{" ready="}{.conditions.ready}{" terminating="}{.conditions.terminating}{"\n"}{end}'

Sample healthy output:

192.168.5.48 ready=true terminating=
192.168.5.42 ready=true terminating=

When one readiness probe fails:

192.168.5.48 ready=false terminating=
192.168.5.42 ready=true terminating=

For a Service with publishNotReadyAddresses: true, Kubernetes treats matching EndpointSlice addresses as ready even when the corresponding Pods are not Ready.

If ENDPOINTS is empty or <none>, check:

• Service spec.selector matches Pod labels
• Pods live in the same namespace as the Service
• The Service is not a headless or manual Endpoints special case you misread

When addresses exist but ready=false, fix readiness probes or terminating Pods instead of changing the selector. Use EndpointSlice commands for inspection. The legacy Endpoints API is deprecated from Kubernetes v1.33 onward.

Compare selectors and Pod labels

Every selector key-value pair must match Pod labels. Selectors match Pod template labels, not Deployment metadata labels outside the Pod template.

Print the Service selector:

kubectl get service web -n svc-trouble-lab -o jsonpath='{.spec.selector}'

Sample output:

{"app":"web"}

List Pod labels in the same namespace:

kubectl get pods -n svc-trouble-lab --show-labels

A Pod-template label change or a wrong Service selector removes all backends. Patch the selector to a non-matching value to reproduce an empty EndpointSlice:

kubectl patch service web -n svc-trouble-lab --type=json -p='[{"op":"replace","path":"/spec/selector/app","value":"web-broken"}]'

Wait until no addresses remain. The EndpointSlice controller continuously evaluates Service selectors and stores matching backends in one or more EndpointSlices, so immediate reads can otherwise show the previous state.

until [ "$(kubectl get endpointslice \
  -n svc-trouble-lab \
  -l kubernetes.io/service-name=web \
  -o jsonpath='{.items[*].endpoints[*].addresses[0]}' |
  wc -w)" -eq 0 ]; do
  sleep 1
done

Re-check EndpointSlices:

kubectl get endpointslice -n svc-trouble-lab -l kubernetes.io/service-name=web

Sample output:

NAME        ADDRESSTYPE   PORTS     ENDPOINTS   AGE
web-k557g   IPv4          <unset>   <none>      45s

Traffic through the Service fails when no addresses are published:

kubectl exec -n svc-trouble-lab net-client -- sh -c '
  curl -sS --max-time 3 http://web:80 >/dev/null
  rc=$?
  echo "curl_exit=$rc"
'

Sample output:

curl: (7) Failed to connect to web port 80
curl_exit=7

A dropped packet path can instead return:

curl: (28) Connection timed out after 3000 milliseconds
curl_exit=28

Restore the correct selector:

kubectl patch service web -n svc-trouble-lab --type=json -p='[{"op":"replace","path":"/spec/selector/app","value":"web"}]'

Wait for both addresses to return:

until [ "$(kubectl get endpointslice \
  -n svc-trouble-lab \
  -l kubernetes.io/service-name=web \
  -o jsonpath='{.items[*].endpoints[*].addresses[0]}' |
  wc -w)" -eq 2 ]; do
  sleep 1
done

After the controller reconciles, Service curls succeed again.

Test readiness removal and recovery

A Pod can be Running and still be NotReady. Normal Service routing uses ready endpoints only.

List Pod readiness:

kubectl get pods -n svc-trouble-lab -l app=web

Remove the readiness marker on one Pod to simulate a failing probe:

kubectl exec -n svc-trouble-lab "$BACKEND_POD" -- rm -f /tmp/healthy

kubectl wait pod/"$BACKEND_POD" -n svc-trouble-lab --for=condition=Ready=false --timeout=60s

Inspect EndpointSlice conditions with the JSONPath command from earlier. EndpointSlice readiness maps to the corresponding Pod readiness condition for Pod-backed endpoints.

Service traffic may still succeed when other ready backends exist:

kubectl exec -n svc-trouble-lab net-client -- curl -s http://web:80

Repeated requests can hit only healthy replicas until readiness recovers. Configure probe YAML in Kubernetes health probes. publishNotReadyAddresses: true is a special override for peer-discovery scenarios, not normal HTTP Services.

Restore readiness before continuing with the port tests:

kubectl exec -n svc-trouble-lab "$BACKEND_POD" -- touch /tmp/healthy

kubectl wait pod/"$BACKEND_POD" -n svc-trouble-lab --for=condition=Ready --timeout=60s

Diagnose Service Ports and the Backend Process

Verify port and targetPort

Clients connect to the Service port. Kubernetes forwards to targetPort on each selected Pod:

Client → Service port 80 → Pod targetPort 80

Confirm the mapping on the Service:

kubectl get service web -n svc-trouble-lab -o jsonpath='port={.spec.ports[0].port} targetPort={.spec.ports[0].targetPort}{"\n"}'

Sample output:

port=80 targetPort=http

Patch targetPort to a port nothing listens on:

kubectl patch service web -n svc-trouble-lab --type=json -p='[{"op":"replace","path":"/spec/ports/0/targetPort","value":8080}]'

kubectl wait endpointslice \
  -n svc-trouble-lab \
  -l kubernetes.io/service-name=web \
  --for=jsonpath='{.ports[0].port}'=8080 \
  --timeout=60s

Service routing fails while direct Pod access on the real port still works:

kubectl exec -n svc-trouble-lab net-client -- sh -c '
  curl -sS --max-time 3 http://web:80 >/dev/null
  rc=$?
  echo "curl_exit=$rc"
'

Sample output:

curl: (7) Failed to connect to web port 80
curl_exit=7

kubectl exec -n svc-trouble-lab net-client -- curl -s -o /dev/null -w "%{http_code}\n" "http://${BACKEND_IP}:80"

Sample output:

200

That split points at targetPort or Service port mapping, not the application process. Restore the named target port:

kubectl patch service web -n svc-trouble-lab --type=json -p='[{"op":"replace","path":"/spec/ports/0/targetPort","value":"http"}]'

kubectl wait endpointslice -n svc-trouble-lab -l kubernetes.io/service-name=web --for=jsonpath='{.ports[0].port}'=80 --timeout=60s

Checklist when ports look correct but traffic still fails:

• Client uses the Service port, not targetPort
• Named targetPort exists on the Pod spec
• Protocol matches (TCP versus UDP)
• Multi-port Services use unique port names

Defining containerPort does not make an application listen on that port.

Test every Pod directly

Bypass the Service and curl each Pod IP plus target port from net-client:

for POD in $(kubectl get pods -n svc-trouble-lab -l app=web -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}'); do

POD_IP=$(kubectl get pod "$POD" -n svc-trouble-lab -o jsonpath='{.status.podIP}')

CODE=$(kubectl exec -n svc-trouble-lab net-client -- curl -s -o /dev/null -w '%{http_code}' --max-time 3 "http://${POD_IP}:80")

  printf '%s %s %s\n' "$POD" "$POD_IP" "$CODE"
done

Sample output:

web-6b5c766787-fhjs2 192.168.5.48 200
web-6b5c766787-nw7mx 192.168.5.42 200

This makes the intermittent-backend diagnosis reproducible rather than checking an arbitrary first Pod.

Interpret Pod IP versus Service results:

When direct Pod access fails, read application logs with kubectl logs, events, and describe before changing the Service.

Confirm the listening address

Traffic arrives at the Pod network namespace IP, not loopback inside the container. A process bound only to 127.0.0.1 is not reachable through the Pod IP or Service.

Inside a backend Pod, list listeners:

kubectl exec -n svc-trouble-lab "$BACKEND_POD" -- netstat -tln

Sample output:

tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN

0.0.0.0:80 accepts connections on the Pod IP. When the image lacks tools, use an ephemeral debug container — see kubectl debug Pods.

Check NetworkPolicy and the Service Data Plane

When Pod IP or Service tests time out instead of returning connection refused, inspect NetworkPolicy on both sides.

• Egress from the client Pod namespace must allow traffic to the backend on the target port
• Ingress to the backend Pod must allow traffic from the client on that port
• Namespace and Pod selectors must match the actual client and server labels

Once both directions are isolated by policy, you need rules on source egress and destination ingress. Configure policies in Kubernetes NetworkPolicy.

Troubleshoot NodePort and LoadBalancer

Confirm ClusterIP routing from net-client before you debug external paths.

NodePort

Expose the Deployment as NodePort only after internal curls succeed:

kubectl expose deployment web -n svc-trouble-lab --name=web-nodeport --type=NodePort --port=80 --target-port=http

Sample output:

service/web-nodeport exposed

Capture its allocated port and a node InternalIP:

NODE_PORT=$(kubectl get service web-nodeport -n svc-trouble-lab -o jsonpath='{.spec.ports[0].nodePort}')

NODE_NAME=$(kubectl get nodes -o jsonpath='{.items[0].metadata.name}')

NODE_IP=$(kubectl get node "$NODE_NAME" -o jsonpath='{.status.addresses[?(@.type=="InternalIP")].address}')

Test from the client Pod:

kubectl exec -n svc-trouble-lab net-client -- curl -s -o /dev/null -w "%{http_code}\n" "http://${NODE_IP}:${NODE_PORT}"

Sample output:

200

With the default externalTrafficPolicy: Cluster, any reachable NodePort address can forward to ready endpoints across the cluster. When externalTrafficPolicy: Local is configured, the selected node must have an eligible local endpoint.

Kubernetes configures the allocated NodePort on every eligible node address and, by default, forwards it to the Service's ready endpoints. Node-local endpoint availability becomes a strict requirement with local external traffic policy.

A successful request from net-client verifies the in-cluster path to <NodeIP>:<NodePort>. It does not verify workstation routing, the node firewall, or any external network between the workstation and node.

Test from the workstation when NODE_IP is routable from it:

curl -sS --connect-timeout 5 -o /dev/null -w '%{http_code}\n' "http://${NODE_IP}:${NODE_PORT}"

Sample output:

200

Run this from the workstation only when NODE_IP is routable from it. If the in-cluster NodePort request succeeds but this request fails, investigate the workstation-to-node route, node firewall, VirtualBox or cloud networking, and any external firewall rules.

LoadBalancer

LoadBalancer troubleshooting belongs after ClusterIP succeeds. Check external address assignment, controller events, health checks, and cloud firewall rules. Keep provider-specific debugging outside this portable workflow. For HTTP host and path routing, continue with expose services with Ingress after the Service path is healthy.

Service Troubleshooting Decision Flow and Symptoms

Can the backend Pod answer on PodIP:targetPort?
├─ No
│  ├─ Check application process and listening address
│  ├─ Check target port
│  ├─ Check Pod readiness and logs
│  └─ Check NetworkPolicy
└─ Yes
   ├─ Does EndpointSlice contain the Pod address?
   │  ├─ No: selector, labels, or namespace
   │  └─ Yes: check ready=false before blaming selector
   ├─ Does ClusterIP:port work?
   │  ├─ No: port mapping or Service data plane
   │  └─ Yes
   ├─ Does Service DNS work?
   │  ├─ No: DNS troubleshooting
   │  └─ Yes
   └─ Does external access work?
      └─ Check NodePort, LoadBalancer or Ingress

What's Next

• Services, CoreDNS and Name Resolution
• Kubernetes NetworkPolicy with Examples
• Kubernetes Ingress Rules with Host, Path and TLS Examples

References

• Kubernetes documentation — Service
• Kubernetes documentation — EndpointSlices
• Kubernetes documentation — DNS for Services and Pods
• Kubernetes documentation — Debug Services

Summary

You troubleshot a Kubernetes Service by working inside out: confirm the application listens on the Pod IP and target port, verify EndpointSlices publish addresses and ready conditions, then test ClusterIP and DNS from a client Pod before touching NodePort or LoadBalancer.

The failures that look alike from a browser often split cleanly in the cluster. Empty EndpointSlice addresses usually mean selector or namespace problems. Addresses with ready=false point at readiness or termination. Pod IP success with Service failure points at port mapping. ClusterIP success with DNS failure belongs in DNS troubleshooting, not Service YAML edits.

Keep a dedicated net-client Pod in the same namespace for reproducible curls. Record Service port, target port, selector, and EndpointSlice output before you patch anything, so you can tell whether the fix restored backends or only masked an external exposure issue.