Kubernetes PodDisruptionBudget Tutorial: Preserve Availability During Node Maintenance

This tutorial demonstrates how to use PodDisruptionBudgets (PDBs) in Kubernetes to prevent downtime during node maintenance operations. We’ll focus exclusively on the PDB mechanism and explore a series of scenarios, iteratively building a solution that handles disruptions gracefully.

Understanding PodDisruptionBudgets

A PodDisruptionBudget (PDB) allows you to limit the number of Pods that can be simultaneously disrupted. This is crucial for maintaining availability when performing operations like node upgrades, draining nodes, or scaling down deployments. Kubernetes will respect the PDB, preventing the disruption of Pods that would violate the defined limit.

Example 1: Initial Setup – Simple PDB

Let’s start with a very basic PDB. We’ll create a deployment and a PDB that allows a maximum of one disruption at a time.

Step 1: Create Deployment

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
  labels:
    app: my-app
spec:
  replicas: 2
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
      - name: my-app-container
        image: nginx:latest
        ports:
        - containerPort: 80

Step 2: Create PodDisruptionBudget

---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: my-app-pdb
spec:
  minAvailable: 1  # Ensure at least one pod is always available
  selector:
    matchLabels:
      app: my-app

Step 3: Verify

kubectl get deployment my-app -o yaml
kubectl get pdb my-app-pdb -o yaml
kubectl describe deployment my-app
kubectl describe pdb my-app-pdb
kubectl get events --sort-by='.lastTimestamp'

Run a command to force a disruption, such as `kubectl drain my-app –delete-local-data=true`. You should see an event indicating that the PDB was respected, preventing the removal of all pods.

Example 2: Introducing a More Complex Scenario – Multiple Deployments

Now let’s extend the scenario to include multiple deployments and a more realistic disruption situation. We’ll introduce a second deployment that needs to be available even during the first deployment’s maintenance.

Step 1: Create the First Deployment

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-app
  labels:
    app: web-app
spec:
  replicas: 2
  selector:
    matchLabels:
      app: web-app
  template:
    metadata:
      labels:
        app: web-app
    spec:
      containers:
      - name: web-app-container
        image: nginx:latest
        ports:
        - containerPort: 80

Step 2: Create the Second Deployment

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-app
  labels:
    app: api-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: api-app
  template:
    metadata:
      labels:
        app: api-app
    spec:
      containers:
      - name: api-app-container
        image: busybox:latest
        command: ["/bin/sh"]
        args: ["-c", "while true; do echo 'API App Running'; sleep 5; done"]
        ports:
        - containerPort: 8080

Step 3: Create the PodDisruptionBudget

---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: web-app-pdb
spec:
  selector:
    matchLabels:
      app: web-app
  minAvailable: 1

Step 4: Verification (Simulate a Drain)

kubectl drain web-app --delete-local-data=true
kubectl get deployments -o yaml
kubectl get pdb web-app-pdb -o yaml
kubectl describe deployment web-app
kubectl get events --sort-by='.lastTimestamp'

Observe that only the `web-app` deployment pods are being drained, respecting the PDB. The `api-app` remains available.

Example 3: Incorrect PDB Configuration – Troubleshooting

Let’s demonstrate what happens when the PDB is misconfigured. We’ll make it allow a disruption of all pods.

Step 1: Modify the PDB

---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: web-app-pdb-incorrect
spec:
  selector:
    matchAllLabels:
      app: web-app
  minAvailable: 0  # Allow all pods to be disrupted

Step 2: Attempt a Drain

kubectl drain web-app --delete-local-data=true
kubectl get deployments -o yaml
kubectl get pdb web-app-pdb-incorrect -o yaml
kubectl describe deployment web-app
kubectl get events --sort-by='.lastTimestamp'

You should see an event indicating that all pods for the `web-app` deployment are being disrupted, demonstrating that the PDB was overridden by the incorrect configuration. `minAvailable: 0` removes the protection.