Kubernetes: Troubleshooting Excessive CPU Usage Across Multiple Pods in a Deployment

This tutorial guides you through troubleshooting excessive CPU usage across multiple pods within a Kubernetes Deployment. We’ll utilize YAML manifests and `kubectl` commands to diagnose and resolve the issue, focusing specifically on the deployment context.

Understanding the Problem

Excessive CPU usage within a Kubernetes deployment can lead to performance bottlenecks, application slowdowns, and ultimately, a poor user experience. It’s crucial to identify which pods are consuming the most CPU and understand why. This guide outlines a systematic approach.

Example 1: Initial Assessment – Identifying High CPU Pods

Let’s start with a simple deployment and observe the CPU usage. We’ll use a basic ‘sleep’ container to simulate CPU-intensive work.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sleep-deployment
  labels:
    app: sleep
spec:
  replicas: 3
  selector:
    matchLabels:
      app: sleep
  template:
    metadata:
      labels:
        app: sleep
    spec:
      containers:
      - name: sleep
        image: busybox
        imagePullPolicy: IfNotPresent
        command: ["/bin/sh", "-c"]
        args: ["while true; do echo 'Hello' >> /tmp/hello.log; sleep 1; done"]
        resources:
          requests:
            cpu: "100m"
          limits:
            cpu: "500m"
        ports:
        - containerPort: 80

Now, deploy this deployment:

kubectl apply -f sleep-deployment.yaml

Check the CPU usage of the pods:

kubectl top pods

Observe the pods labeled `sleep-deployment`. You’ll likely see a high CPU usage for all three. This confirms the initial problem – all pods are consuming significant CPU. This is a good starting point for investigation.

Example 2: Resource Limits and Requests – Fine-tuning the Deployment

The initial setup uses requests and limits. The `100m` request means the pods are guaranteed at least 100 millicores, while the `500m` limit means they can’t exceed 500 millicores. Let’s reduce the request to see if it impacts usage. We’ll also introduce a resource quota.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sleep-deployment-refined
  labels:
    app: sleep
spec:
  replicas: 3
  selector:
    matchLabels:
      app: sleep
  template:
    metadata:
      labels:
        app: sleep
    spec:
      containers:
      - name: sleep
        image: busybox
        imagePullPolicy: IfNotPresent
        command: ["/bin/sh", "-c"]
        args: ["while true; do echo 'Hello' >> /tmp/hello.log; sleep 1; done"]
        resources:
          requests:
            cpu: "50m" # Reduced CPU request
          limits:
            cpu: "250m" # Reduced CPU limit
        ports:
        - containerPort: 80
      resources:
        requests:
          memory: "64Mi"
        limits:
          memory: "256Mi"
      metadata:
        annotations:
          container.kubernetes.io/limits: cpu=250m,memory=256Mi

Apply the updated deployment:

kubectl apply -f sleep-deployment-refined.yaml

Re-run `kubectl top pods`. You should observe a reduction in CPU usage across all pods. This demonstrates the impact of carefully setting resource requests and limits.

Example 3: Examining Pod Logs and Events

If reducing resources doesn’t fully resolve the issue, we need to investigate the application’s behavior. Let’s examine the logs and events for the pods.

kubectl logs sleep-deployment-refined -n default --all-containers

This command fetches logs from all containers in all pods of the `sleep-deployment-refined` deployment within the `default` namespace. Carefully examine the output for errors, warnings, or unexpected behavior. Look for any patterns that might be causing excessive CPU usage (e.g., infinite loops, inefficient algorithms).

Also, check for events related to the pods:

kubectl describe events -n default --selector=app=sleep

This command lists events related to pods with the label `app=sleep`. Look for events like “OOMKilled” (Out of Memory Killed), which indicates the pod exceeded its memory limit and was terminated, often leading to CPU spikes as the pod restarts. Pay attention to events indicating errors or warnings.

If errors are present in the logs or events indicate an issue within the container, you can investigate further within the container itself (e.g., using `kubectl exec` to run commands and diagnose the problem).