Trobleshooting: Quick Visual Reference

Troubleshooting in Kubernetes can be a daunting task. Here is a comprehensive diagram to help you visually debug your deployments in Kubernetes.

Complete Kubernetes Deployment Troubleshooting Flowchart

When you wish to deploy an application in Kubernetes, you usually define three core components:

• A Deployment: A blueprint/recipe for creating and maintaining replicas of your application Pods.
• A Service: An internal load balancer that routes network traffic to target Pods.
• An Ingress: A specification of how external HTTP/HTTPS traffic from outside the cluster flows into your internal Service.

Here is a quick visual recap of how these three layers interact:

Visual Breakdown: Kubernetes Exposure Architecture

1. Dual Load Balancer Layers: Applications are exposed via internal and external load balancers.

2. Service & Ingress: The internal load balancer is the Service; the external entry point is the Ingress.

3. Deployment Controller: Pods are not managed directly. Deployments spin up Pods and maintain replica count.

Assuming you wish to deploy a simple Hello World application, the YAML manifest for such an application looks like this:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-deployment
  labels:
    track: canary
spec:
  selector:
    matchLabels:
      any-name: my-app
  template:
    metadata:
      labels:
        any-name: my-app
    spec:
      containers:
        - name: cont1
          image: ghcr.io/learnk8s/app:1.0.0
          ports:
            - containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: my-service
spec:
  ports:
    - port: 80
      targetPort: 8080
  selector:
    name: app
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-ingress
spec:
  rules:
  - http:
      paths:
      - backend:
          service:
            name: my-service
            port:
              number: 80
        path: /
        pathType: Prefix

This definition is quite lengthy, and it is easy to overlook how the components relate to one another. Common questions arise instantly:

• When should you use port 80, and when should you use port 8080?
• Should you assign a unique port number for every Service so they do not conflict?
• Do label names matter? Should they be identical everywhere in the manifest?

Before jumping into troubleshooting steps, let's establish how these three components link together.

1 Connecting Deployment and Service

To successfully link a Service to your Pods, pay close attention to three fundamental rules:

1. The Service selector must match at least one label defined in the Pod template.
2. The Service's targetPort must match the containerPort exposed inside the container.
3. The Service port can be any integer. Multiple Services can share port 80 because each Service gets a unique Cluster IP address assigned.

Visualizing Port Mapping (Deployment & Service)

1. Pod exposed by Service

2. Define containerPort

3. Service port vs targetPort

4. targetPort = containerPort

5. Example: Container port 3000

Correcting the labels and ports alignment in our earlier YAML yields:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-deployment
  labels:
    track: canary
spec:
  selector:
    matchLabels:
      any-name: my-app
  template:
    metadata:
      labels:
        any-name: my-app
    spec:
      containers:
        - name: cont1
          image: ghcr.io/learnk8s/app:1.0.0
          ports:
            - containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: my-service
spec:
  ports:
    - port: 80
      targetPort: 8080
  selector:
    any-name: my-app

How to test this connection? Verify that Pods carry the correct labels using kubectl:

kubectl get pods --show-labels
# Output:
# NAME                  READY   STATUS    LABELS
# my-deployment-pv6pd   1/1     Running   any-name=my-app,pod-template-hash=7d6979fb54
# my-deployment-f36rt   1/1     Running   any-name=my-app,pod-template-hash=7d6979fb54

# Filter pods by specific selector label:
kubectl get pods --selector any-name=my-app --show-labels

You can test connectivity to your Service using kubectl port-forward:

kubectl port-forward service/my-service 3000:80
# Forwarding from 127.0.0.1:3000 -> 8080
# Forwarding from [::1]:3000 -> 8080

Where 3000 is your local computer's port, and 80 is the Service port. If you get a valid response at http://localhost:3000, your Service-to-Pod routing is functioning correctly!

2 Connecting Service and Ingress

The next layer involves exposing your app to external cluster clients by configuring the Ingress resource.

Two essential fields must align between the Ingress and the Service:

1. The Ingress service.name must exactly match the Service metadata.name.
2. The Ingress service.port.number must match the Service spec.ports[].port.

Visualizing Ingress to Service Linkage

1. Service exposes port

2. Ingress specifies servicePort

3. Service port & Ingress port match

4. Example: Match on port 80

apiVersion: v1
kind: Service
metadata:
  name: my-service
spec:
  ports:
    - port: 80
      targetPort: 8080
  selector:
    any-name: my-app
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-ingress
spec:
  rules:
  - http:
      paths:
      - backend:
          service:
            name: my-service
            port:
              number: 80
        path: /
        pathType: Prefix

Testing Ingress Routing: Forward ports directly to your Ingress Controller Pod:

# 1. Locate Ingress Controller pod
kubectl get pods --all-namespaces

# 2. Inspect exposed ports on the Ingress pod
kubectl describe pod nginx-ingress-controller-6fc5bcc --namespace kube-system | grep Ports
# Output: Ports: 80/TCP, 443/TCP, 18080/TCP

# 3. Port forward to the Ingress controller
kubectl port-forward nginx-ingress-controller-6fc5bcc 3000:80 --namespace kube-system

3 3 Steps to Troubleshoot Kubernetes Deployments

Having a structured mental model is essential when dealing with broken deployments. Always debug starting from the bottom of the stack upwards:

The Bottom-Up Debugging Model

Step 1: Debug Pods

Verify that Pods are in Running and Ready status.

Step 2: Debug Service

Confirm Service routes traffic to healthy Pod endpoints.

Step 3: Debug Ingress

Examine connection between Ingress controller and Service.

Step 1: Troubleshooting Pods

Most deployment failures stem from issues within the Pod itself. First check Pod status:

kubectl get pods
# NAME                    READY STATUS            RESTARTS  AGE
# app1                    0/1   ImagePullBackOff  0         47h
# app2                    0/1   Error             0         47h
# app3-76f9fcd46b-xbv4k   1/1   Running           1         47h

Four primary commands provide key insights into failing Pods:

• kubectl logs <pod-name> — Retrieves stdout/stderr logs from container(s).
• kubectl describe pod <pod-name> — Displays lifecycle events, conditions, and resource states.
• kubectl get pod <pod-name> -o yaml — Extracts the active YAML definition stored in etcd.
• kubectl exec -ti <pod-name> -- bash — Opens an interactive shell inside a running container.

Catalog of Common Pod Errors

Deep-Dive & Solutions for Top Pod Errors:

1. ImagePullBackOff / ErrImagePull:

   Occurs when Kubernetes cannot fetch the container image. Top culprits:

   • Misspelled image name or non-existent image repository.
   • Specified tag does not exist.
   • Image resides in a private registry and imagePullSecrets are missing.

   Fix: Correct image name/tag, or add docker-registry secret credentials and reference in Pod spec.

2. CrashLoopBackOff:

   The container starts, fails, and continuously restarts. Caused by application crashes, misconfiguration, or failing liveness probes.

   Fix: Retrieve logs from the previous crashed container instance:

   kubectl logs <pod-name> --previous
3. RunContainerError:

   Container fails before the application inside starts executing. Caused by mounting non-existent ConfigMaps/Secrets or mounting read-only volumes as read-write.

   Fix: Inspect event logs using kubectl describe pod <pod-name>.

4. Pod Pending State:

   Pod remains stuck in Pending state without being scheduled onto a node. Causes:

   • Insufficient CPU/Memory resources on cluster nodes.
   • Namespace exceeded its defined ResourceQuota object.
   • Pod bound to a Pending PersistentVolumeClaim.

   Fix: Inspect Events section in describe output, or view cluster-wide events:

   kubectl get events --sort-by=.metadata.creationTimestamp
5. Pod Not Ready State:

   Pod status is Running, but 0/1 READY. Indicates that the Readiness probe is failing. The Service will NOT route traffic to this Pod.

   Fix: Check readiness probe path/port and examine kubectl describe pod events.

Step 2: Troubleshooting Services

If Pods are Running and Ready, but the application remains unreachable, inspect the Service layer.

Services route traffic to Pods via Endpoints. Verify that your Service has active target Pod IP endpoints:

kubectl describe service my-service
# Name:                     my-service
# Namespace:                default
# Selector:                 any-name=my-app
# IP:                       10.100.194.137
# Port:                     80/TCP
# TargetPort:               8080/TCP
# Endpoints:                172.17.0.5:8080, 172.17.0.6:8080

Step 3: Troubleshooting Ingress

If Pods are Ready and Service endpoints are active, but external requests fail, debug the Ingress resource.

Check the backend linkage of your Ingress resource:

kubectl describe ingress my-ingress
# Name:             my-ingress
# Namespace:        default
# Rules:
#   Host        Path  Backends
#   ----        ----  --------
#   *           /     my-service:80 (172.17.0.5:8080, 172.17.0.6:8080)

If the Backends column displays <error: endpoints "my-service" not found>, the service.name or service.port in your Ingress definition is misconfigured.

Debugging NGINX Ingress Controller

If using NGINX Ingress Controller, install and utilize the official kubectl ingress-nginx CLI plugin for fast debugging:

# Validate NGINX configuration syntax
kubectl ingress-nginx lint --namespace kube-system

# Inspect backends configured inside NGINX
kubectl ingress-nginx backend --namespace kube-system

# Tail NGINX Ingress controller logs
kubectl ingress-nginx logs --namespace kube-system

Summary & Key Takeaways

Special thanks to Gergely Risko, Daniel Weibel, and Charles Christyraj for invaluable contributions to this visual guide.