Kubernetes Roll Back Deployment After Readiness Probe Causes a Failed Rollout

This tutorial demonstrates how to roll back a Kubernetes Deployment after a Readiness Probe failure causes a failed rollout. We’ll walk through a common scenario and use kubectl commands to diagnose and resolve the issue.

Scenario: Readiness Probe Issues

A common cause of failed rollouts is an issue with the Deployment’s Readiness Probe. If the probe continuously fails, Kubernetes will not consider the Deployment healthy and won’t automatically update. This tutorial focuses on the process of identifying and correcting this issue, culminating in a rollback.

Example 1: Initial Rollout and Failure

Let’s set up a simple Deployment and intentionally trigger a Readiness Probe failure to observe the rollback process.

kubectl create deployment my-app --image=nginx:latest --replicas=3
kubectl expose deployment my-app --port=80
kubectl rollout status deployment my-app

After executing these commands, verify the Deployment’s status:

kubectl get deployments my-app -o wide

Observe the ‘READY’ column. It should indicate that only 1 or 2 pods are ready, as the Readiness Probe is failing. The output will also show events related to the rollout.

kubectl get deployments my-app -o wide
kubectl get pods -o wide
kubectl get events -n default --sort-by='.lastTimestamp'

The events will likely show messages related to the Readiness Probe failing for one or more pods. Note the event timestamps and messages.

Example 2: Identifying the Problem and Correcting the Probe

Now, let’s assume the Readiness Probe is failing because the Nginx server isn’t immediately accepting connections. We need to adjust the probe configuration.

kubectl edit deployment my-app

In the YAML editor, modify the `readinessProbe` section of the Deployment to increase the delay before the probe attempts a connection:

spec:
  template:
    spec:
      containers:
      - name: my-app
        image: nginx:latest
        ports:
        - containerPort: 80
        readinessProbe:
          httpGet:
            path: /
            port: 80
          initialDelaySeconds: 30
          periodSeconds: 10
          timeoutSeconds: 5

Save the changes. Kubernetes will automatically update the Deployment. Verify the changes:

kubectl rollout restart deployment my-app
kubectl get deployments my-app -o wide
kubectl get pods -o wide

The Readiness Probe should now be successful, and all three pods should report as ready. Observe the events again to confirm the Readiness Probe is passing.

Example 3: Rollback on a Failed Update

Let’s intentionally create a problem and trigger a rollback.

kubectl set image deployment my-app nginx:wrong-version --replicas=3

This command updates the image to a non-existent version, causing the Deployment to fail the rollout. Check the rollout status:

kubectl rollout status deployment my-app

You’ll see an error message indicating the rollout is in a failed state. Examine the events:

kubectl get events -n default --sort-by='.lastTimestamp'

The events will show the failed rollout and the reasons (e.g., probe failure, image pull error). To roll back, use:

kubectl rollout undo deployment my-app

Kubernetes will automatically revert the Deployment to its previous state. Verify the rollback:

kubectl rollout status deployment my-app
kubectl get deployments my-app -o wide

The Deployment should now be back to the previous version, and the pods should be ready again. The events will reflect the rollback action.