Kubernetes Rollback Tutorial: Readiness Probe Failure

This tutorial demonstrates how to roll back a Kubernetes Deployment after a Readiness Probe failure during a rollout. This is a common scenario when an application experiences issues that affect its ability to respond to traffic.

Scenario

We’ll create a Deployment that repeatedly fails its readiness probe due to a simulated application issue. Then, we’ll use `kubectl rollout undo` to revert to a previous, stable version of the Deployment.

Example 1: Initial Deployment & Failure

This example establishes the initial deployment and demonstrates the readiness probe failing, triggering the rollback.

Step 1: Create the Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app-deployment
  labels:
    app: my-app
spec:
  replicas: 3
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
      - name: my-app-container
        image: busybox:latest
        ports:
        - containerPort: 8080
        readinessProbe:
          httpGet:
            path: /
            port: 8080
          initialDelaySeconds: 15
          periodSeconds: 10
          timeoutSeconds: 5
          failureThreshold: 3
          successThreshold: 1

Run this YAML using `kubectl apply -f my-app-deployment.yaml`. You should see output confirming the deployment is created.

Step 2: Observe the Failure

kubectl get deployments my-app-deployment -o wide

Initially, you’ll see the Deployment’s strategy is “RollingUpdate,” and the number of available replicas will increase. After a few seconds, observe the status. You should see a “Progressing” status for a few pods, followed by a “NotReady” status for some pods. Continue to monitor the pods with `kubectl get pods -l app=my-app`. You should observe the readiness probe failing repeatedly for specific pods.

Examine the events related to the Deployment:

kubectl get events -n default --selector app=my-app

You will likely see messages related to the readiness probe failing and pods being marked as “NotReady.”

Example 2: Rolling Back the Deployment

This example demonstrates the use of `kubectl rollout undo` to revert to a previous stable version.

Step 1: Initiate the Rollback

kubectl rollout undo deployment my-app-deployment

This command initiates the rollback process. It will typically roll back to the previous revision of the Deployment.

Step 2: Verify the Rollback

kubectl get deployments my-app-deployment -o wide

The Deployment’s strategy should now be “RollingBack.” The number of available replicas will decrease as pods are terminated and replaced with the previous version. Continue to observe the pods’ status with `kubectl get pods -l app=my-app` to confirm the pods are transitioning back to “Ready” status.

Check the events again:

kubectl get events -n default --selector app=my-app

You should see events indicating the rollback operation is in progress.

Example 3: Completing the Rollback & Recovery

This example shows the final state of the deployment after the rollback is complete.

Step 1: Confirm the Rollback is Complete

kubectl get deployments my-app-deployment -o wide

The Deployment’s strategy should revert to “RollingUpdate,” and the number of available replicas will return to the original value (3 in this case). The pods should transition back to “Ready” status.

Verify the readiness probe is passing now:

kubectl get pods my-app-container -o jsonpath='{.status.ready}'

The output should be “true” for all pods.

Finally, examine the events one last time:

kubectl get events -n default --selector app=my-app

You should see events indicating that the deployment has successfully transitioned back to a stable state.