Kubernetes Rolling Update Stuck in a Failed Rollout State

This tutorial guides you through diagnosing and resolving a common Kubernetes issue: a rolling update that gets stuck in a “Failed Rollout” state. We’ll use declarative YAML manifests and `kubectl` commands to demonstrate the process, focusing exclusively on Kubernetes.

Understanding the Problem

A rolling update aims to upgrade applications without downtime. When a rollout fails, Kubernetes attempts to automatically retry the failed deployments. However, these retries can sometimes get stuck, leading to a persistent “Failed Rollout” status. This situation requires investigation to identify the root cause, which could range from resource constraints to application errors.

Example 1: Initial Observation and Basic Diagnostics

Let’s start with a basic scenario where a rolling update has unexpectedly failed. Assume we have a Deployment named my-app using a ReplicaSet named my-app-rs. We’ll examine the rollout status and events to gain initial insights.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
  labels:
    app: my-app
spec:
  replicas: 3
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
      - name: my-app-container
        image: nginx:latest
        ports:
        - containerPort: 80

Now, execute the following commands:

kubectl rollout status deployment/my-app
kubectl get deployments -n default my-app
kubectl get replicasets -n default my-app-rs
kubectl get pods -n default -l app=my-app
kubectl describe rollout status deployment/my-app

Typical output (after a brief failure):

deployment/my-app status: Initial rollout succeeded, but existing resources have been lost.
deployment.apps/my-app
  AVAILABLE   3/3
replicaset.apps/my-app-rs
  READY      2/2
  UP         2/2
pods -n default (1 in total)
  my-app-7969d746dd-8mvkx   1/1     Running   (0pts, 0io)   nginx:latest
kubectl describe rollout status deployment/my-app
...
Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Failed   FailedDeployment  2m    my-app-controller  Deployment "my-app" failed to roll out.

The output reveals that the rollout failed and the controller encountered an issue. The events section is crucial – it will usually provide hints about the problem (e.g., container failure, image pull errors). Note the `FailedDeployment` event.

Example 2: Investigating Container Issues

Let’s assume the `kubectl describe rollout status` output indicates a problem with the container image pull. We’ll examine the pod logs to confirm the error.

kubectl logs my-app-7969d746dd-8mvkx -n default

Possible Log Output (image pull failure):

Failed to pull image "nginx:latest": rpc error: code = Unknown desc = Error response from daemon: Error status 500, Internal error processing create request for container "my-app-container": timed out waiting for the container image "nginx:latest" to become present

This log confirms the image pull failed due to a timeout. This often points to network connectivity problems, DNS resolution issues, or registry access restrictions.

Example 3: Addressing the Image Pull Issue

Let’s assume the network issue was a DNS problem. We’ll temporarily specify a direct image URL to bypass the registry. This is a drastic measure for troubleshooting only.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
  labels:
    app: my-app
spec:
  replicas: 3
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
      - name: my-app-container
        image: nginx:latest  # Replace with direct image URL for troubleshooting
        ports:
        - containerPort: 80

kubectl rollout restart deployment/my-app
kubectl rollout status deployment/my-app
kubectl get pods -n default -l app=my-app
kubectl describe rollout status deployment/my-app

After restarting the deployment with the direct image URL, the rollout should complete successfully. Once the root cause of the DNS issue is identified and resolved (e.g., updating DNS records), the direct image URL should be removed from the manifest for production deployments.