Kubernetes Roll Back Tutorial: Readiness Probe Failure

This tutorial demonstrates how to roll back a Kubernetes Deployment after a readiness probe failure causes a failed rollout. We’ll use YAML manifests and `kubectl` commands to step through the process. The goal is to understand how to quickly recover from a deployment issue without needing to rebuild the application.

Example 1: Initial Deployment and Failure

We’ll start with a simple Deployment and intentionally configure the readiness probe to fail. This simulates a common scenario where an application starts up but doesn’t immediately report being ready.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app-deployment
  labels:
    app: my-app
spec:
  replicas: 3
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
      - name: my-app-container
        image: nginx:latest
        ports:
        - containerPort: 80
        readinessProbe:
          httpGet:
            path: /
            port: 80
          initialDelaySeconds: 5
          periodSeconds: 10
          failureThreshold: 3
        resources:
          requests:
            cpu: "100m"
            memory: "128Mi"
          limits:
            cpu: "200m"
            memory: "256Mi"

Steps:

1. Create the Deployment manifest (e.g., `my-app-deployment.yaml`).
2. Apply the manifest: `kubectl apply -f my-app-deployment.yaml`
3. Observe the rollout: `kubectl get deployments my-app-deployment`
4. Check the status: `kubectl describe deployment my-app-deployment` – You should see the deployment in a `Failed` state.

Example 2: Identifying and Correcting the Readiness Probe

Now, we’ll examine the problem and adjust the readiness probe configuration to allow the deployment to continue. The issue is likely that the probe is failing consistently because it’s checking the root path of the nginx image, which isn’t immediately ready.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app-deployment
  labels:
    app: my-app
spec:
  replicas: 3
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
      - name: my-app-container
        image: nginx:latest
        ports:
        - containerPort: 80
        readinessProbe:
          httpGet:
            path: /nginx-health
            port: 80
          initialDelaySeconds: 5
          periodSeconds: 10
          failureThreshold: 3
        resources:
          requests:
            cpu: "100m"
            memory: "128Mi"
          limits:
            cpu: "200m"
            memory: "256Mi"

Changes:

• We modified the `readinessProbe.httpGet.path` to `/nginx-health`.
• We added the `/nginx-health` file to the Nginx image, returning a 200 OK response. This simulates the application becoming ready.

Steps:

1. Modify the Deployment manifest (e.g., `my-app-deployment.yaml`) with the change.
2. Apply the manifest: `kubectl apply -f my-app-deployment.yaml`
3. Observe the rollout: `kubectl get deployments my-app-deployment` – The status should change to `Progressing`.
4. Check the status: `kubectl describe deployment my-app-deployment` – The deployment should now be in a `Running` state.

Example 3: Rolling Back After a Misconfiguration (Simulated)

Let’s intentionally introduce a new issue – a change to the readiness probe that causes it to fail again. This demonstrates a rollback scenario.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app-deployment
  labels:
    app: my-app
spec:
  replicas: 3
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
      - name: my-app-container
        image: nginx:latest
        ports:
        - containerPort: 80
        readinessProbe:
          httpGet:
            path: /incorrect-health-check
            port: 80
          initialDelaySeconds: 5
          periodSeconds: 10
          failureThreshold: 3
        resources:
          requests:
            cpu: "100m"
            memory: "128Mi"
          limits:
            cpu: "200m"
            memory: "256Mi"

Steps:

1. Modify the Deployment manifest (e.g., `my-app-deployment.yaml`) to change the readiness probe path to `/incorrect-health-check`.
2. Apply the manifest: `kubectl apply -f my-app-deployment.yaml`
3. Observe the rollout: `kubectl get deployments my-app-deployment` – The status will revert to `Failed`.
4. Check the status: `kubectl describe deployment my-app-deployment` – The deployment should be in `Failed` state.
5. To roll back, revert the change to the original readiness probe path (e.g., `/nginx-health`) and apply the updated manifest.

Note: This tutorial focuses specifically on rolling back a Deployment due to a Readiness Probe failure. More complex rollback strategies involving multiple Deployments and services require additional consideration.