Kubernetes: Investigate Excessive CPU Usage in a StatefulSet via ResourceQuotas

Kubernetes: Investigating Excessive CPU Usage in a StatefulSet via ResourceQuotas

This tutorial focuses on identifying and mitigating excessive CPU usage within a Kubernetes StatefulSet, utilizing ResourceQuotas for granular control. We’ll walk through a practical scenario, examining common causes and employing diagnostic tools to pinpoint the issue and implement solutions.

Scenario: Excessive CPU Usage in a Database StatefulSet

We’ll simulate a scenario where a PostgreSQL StatefulSet running in a Kubernetes cluster is consistently consuming excessive CPU resources. This can be due to various factors, including resource constraints, inefficient queries, or application-level issues. Our goal is to use ResourceQuotas to limit the resources available to the StatefulSet and then investigate the logs and metrics to determine the cause.

Example 1: Initial Setup and ResourceQuota Creation

First, let’s create a basic PostgreSQL StatefulSet and then introduce a ResourceQuota to limit its CPU resources.


apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgresql-statefulset
spec:
  serviceName: postgresql-service
  replicas: 3
  selector:
    matchLabels:
      app: postgresql
  template:
    metadata:
      labels:
        app: postgresql
    spec:
      containers:
      - name: postgresql
        image: postgres:15
        resources:
          requests:
            cpu: "100m"
          limits:
            cpu: "500m"
---
apiVersion: v1
kind: Service
metadata:
  name: postgresql-service
spec:
  selector:
    app: postgresql
  ports:
  - protocol: TCP
    port: 5432
    targetPort: 5432
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: postgresql-reader
  namespace: default
rules:
- apiGroups: [""]
  verbs: ["get", "list"]
  resources: ["pods"]
- apiGroups: [""]
  verbs: ["get", "list"]
  resources: ["services"]
- apiGroups: [""]
  verbs: ["get", "list"]
  resources: ["configmaps"]


kubectl apply -f statefulset_with_resourcequota.yaml
This creates a PostgreSQL StatefulSet with three replicas, each requesting 100m of CPU and having a limit of 500m.  We also create a service and a minimal Role for basic access.
Example 2: Introducing a ResourceQuota and Monitoring CPU Usage
Now, let’s introduce a ResourceQuota to limit the total CPU resources the StatefulSet can consume. We'll then monitor CPU usage to see the effect.

apiVersion: v1
kind: ResourceQuota
metadata:
  name: postgresql-quota
  namespace: default
spec:
  hard:
    cpu: "1"

kubectl apply -f resourcequota.yaml
This creates a ResourceQuota named `postgresql-quota` in the `default` namespace, limiting the total CPU resources to 1. This will severely constrain the StatefulSet's resource consumption.
kubectl describe resourcequota postgresql-quota
Observe the output. Pay attention to the `Used` field. This shows how much CPU the StatefulSet has actually consumed. If the `Used` value is significantly exceeding the `hard` limit of 1, it indicates a problem with the StatefulSet’s workload or configuration.
kubectl top pods --all-namespaces | grep postgresql
This command displays the CPU usage of all pods in all namespaces. Examine the output for pods labeled with "app: postgresql" to see the actual CPU consumption.  If the CPU usage is consistently high, it confirms the issue. 
Example 3: Investigating Logs and Correcting the Problem (Simulated)
Let's assume that after monitoring, you discover that the excessive CPU usage is due to a poorly optimized PostgreSQL query that's running repeatedly. We'll simulate this by adding a delay to a query to artificially increase CPU usage.

# Simulate a slow query by introducing a delay in the postgresql container's command.
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgresql-statefulset
spec:
  serviceName: postgresql-service
  replicas: 3
  selector:
    matchLabels:
      app: postgresql
  template:
    metadata:
      labels:
        app: postgresql
    spec:
      containers:
      - name: postgresql
        image: postgres:15
        command: ["/usr/bin/pg_activity"]
        args: ["-c","SELECT sleep(5)"] # Introducing a 5-second sleep

kubectl apply -f statefulset_with_slow_query.yaml
Now, run kubectl top pods --all-namespaces | grep postgresql again. You should see the CPU usage of the PostgreSQL pods significantly increased due to the delayed query. The ResourceQuota will limit the overall CPU consumption, preventing the StatefulSet from exceeding its allocated limit.
To resolve the issue, you'd then optimize the PostgreSQL query to remove the unnecessary delay.  Alternatively, you could adjust the ResourceQuota limit if the query is a legitimate workload requiring more CPU.