Kubernetes: Stopping OOMKilled Restarts by Correcting Requests and Limits

This tutorial focuses on preventing your Kubernetes pods from being killed due to Out Of Memory (OOM) conditions. The common solution is to adjust the resource requests and limits defined in your pod specifications. We’ll explore this through practical examples, demonstrating how to identify OOMKilled events and how to correctly configure your deployments.

Understanding OOMKilled Events

When a pod exceeds its memory limits, Kubernetes automatically terminates it. This is known as an OOMKilled event. Kubernetes logs these events, providing valuable information for troubleshooting.

# Example of an OOMKilled Event (from kubectl events)
# ...
involvedObject.kind: Pod
involvedObject.name: my-pod
involvedObject.namespace: default
reason: OOMKilled
message: "MemoryPressure"
...

The “MemoryPressure” reason indicates the pod was actively exceeding its memory limit. It’s crucial to understand why this is happening.

Example 1: Initial Deployment – Immediate OOMKilled

We’ll start with a simple deployment that immediately triggers an OOMKilled event because it lacks resource limits. This demonstrates the core problem.

Step 1: Create the Deployment Manifest

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-deployment
  labels:
    app: my-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
      - name: my-container
        image: busybox:latest
        resources:
          requests:
            memory: "64Mi"
          limits:
            memory: "64Mi"

Step 2: Apply the Manifest

kubectl apply -f my-deployment.yaml

Step 3: Observe the OOMKilled Event

kubectl get events --namespace default --watch --sort-by='.lastTimestamp'

You should quickly see OOMKilled events appearing in the output. This confirms the initial problem – the `busybox` container is attempting to use more memory than it’s allocated.

Example 2: Adding Resource Limits – Preventing OOMKilled

Now, let’s add resource limits to the deployment to prevent the OOMKilled event. This demonstrates the corrective action.

Step 1: Modify the Deployment Manifest

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-deployment
  labels:
    app: my-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
      - name: my-container
        image: busybox:latest
        resources:
          requests:
            memory: "64Mi"
          limits:
            memory: "128Mi"

Step 2: Apply the Updated Manifest

kubectl apply -f my-deployment.yaml

Step 3: Verify the Change

kubectl describe deployment my-deployment --namespace default

Check the `Limits` section in the output. You’ll see that the memory limit has been increased to 128Mi. Observe that the OOMKilled events should disappear, or significantly reduce in frequency.

Example 3: Correcting Requests and Limits – A More Realistic Scenario

Let’s simulate a slightly more realistic scenario. We’ll use a container that legitimately needs more memory under load. We’ll adjust both the requests and limits to match the observed behavior.

Step 1: Create a Deployment for a Simple Web Server

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-web-deployment
  labels:
    app: my-web-app
spec:
  replicas: 2
  selector:
    matchLabels:
      app: my-web-app
  template:
    metadata:
      labels:
        app: my-web-app
    spec:
      containers:
      - name: my-web-container
        image: nginx:latest
        resources:
          requests:
            memory: "128Mi"
          limits:
            memory: "256Mi"

Step 2: Apply the Manifest

kubectl apply -f my-web-deployment.yaml

Step 3: Monitor Resource Usage (using kubectl top)

kubectl top pods --namespace default

Observe the memory usage of the `my-web-container` pods. The application is actively utilizing memory, confirming the need for the 256Mi limit. If the limit is set too low, this pod will OOMKilled. Adjust appropriately.

By understanding and correctly configuring requests and limits, you can effectively prevent OOMKilled restarts in your Kubernetes deployments, ensuring stability and performance.