Kubernetes Startup Probe Troubleshooting: Resolving Initialization Delays

Startup probes in Kubernetes are designed to ensure your applications are ready to serve traffic before they’re considered healthy. However, if your application takes a significant amount of time to initialize (e.g., database connections, JVM warm-up), the startup probe will repeatedly fail, causing your pod to remain in a `Pending` or `CrashLoopBackOff` state. This tutorial guides you through diagnosing and resolving these issues using Kubernetes manifests and commands.

Understanding the Problem

A startup probe periodically performs a readiness check on a pod. If the probe fails, Kubernetes restarts the pod. When an application takes a while to initialize, the probe repeatedly fails before the application is truly ready, leading to a problematic feedback loop. We’ll focus on situations where the application initialization is genuinely the root cause, not transient network issues or other factors.

Example 1: Simple Delayed Application – Initial Failure

This example demonstrates a basic scenario where a simple application intentionally delays initialization to trigger a startup probe failure.

Manifest (app-delayed.yaml)

apiVersion: apps/v1
kind: Deployment
metadata:
  name: delayed-app
  labels:
    app: delayed-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: delayed-app
  template:
    metadata:
      labels:
        app: delayed-app
    spec:
      containers:
      - name: delayed-app
        image: busybox
        command: ["/bin/sh", "-c", "echo 'Starting...'; sleep 10; echo 'Ready!';"]
        ports:
        - containerPort: 80
        readinessProbe:
          httpGet:
            path: /ready
            port: 80
          initialDelaySeconds: 15
          periodSeconds: 10
          timeoutSeconds: 5
        resources:
          requests:
            cpu: "100m"
            memory: "64Mi"
          limits:
            cpu: "500m"
            memory: "128Mi"

kubectl Commands

kubectl apply -f app-delayed.yaml
kubectl get deployments delayed-app
kubectl get pods -l app=delayed-app
kubectl logs delayed-app-7w8d6q6b -f

Expected Outcome

Initially, the pod will be in `Pending` status. After the 10-second sleep, the container will output “Ready!”, but the startup probe will immediately fail because the `initialDelaySeconds` (15 seconds) is greater than the sleep time. The pod will then restart repeatedly. Examine the logs using `kubectl logs` to see the repeated “Starting…” and “Ready!” output.

Example 2: Adjusting Initial Delay – Success

Now, we’ll modify the `initialDelaySeconds` to align with the sleep duration, resolving the startup probe failure.

Manifest (app-adjusted.yaml)

apiVersion: apps/v1
kind: Deployment
metadata:
  name: delayed-app
  labels:
    app: delayed-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: delayed-app
  template:
    metadata:
      labels:
        app: delayed-app
    spec:
      containers:
      - name: delayed-app
        image: busybox
        command: ["/bin/sh", "-c", "echo 'Starting...'; sleep 10; echo 'Ready!';"]
        ports:
        - containerPort: 80
        readinessProbe:
          httpGet:
            path: /ready
            port: 80
          initialDelaySeconds: 10
          periodSeconds: 10
          timeoutSeconds: 5
        resources:
          requests:
            cpu: "100m"
            memory: "64Mi"
          limits:
            cpu: "500m"
            memory: "128Mi"

kubectl Commands

kubectl apply -f app-adjusted.yaml
kubectl get deployments delayed-app
kubectl get pods -l app=delayed-app
kubectl logs delayed-app-7w8d6q6b -f

Expected Outcome

The pod should now be in the `Running` state. The startup probe will pass successfully after the initial 10-second delay. Verify this by checking the pod status and the logs.

Example 3: Increasing Period – Ongoing Monitoring

Let’s increase the `periodSeconds` to 20, ensuring the probe checks more frequently.

Manifest (app-period.yaml)

apiVersion: apps/v1
kind: Deployment
metadata:
  name: delayed-app
  labels:
    app: delayed-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: delayed-app
  template:
    metadata:
      labels:
        app: delayed-app
    spec:
      containers:
      - name: delayed-app
        image: busybox
        command: ["/bin/sh", "-c", "echo 'Starting...'; sleep 10; echo 'Ready!';"]
        ports:
        - containerPort: 80
        readinessProbe:
          httpGet:
            path: /ready
            port: 80
          initialDelaySeconds: 10
          periodSeconds: 20
          timeoutSeconds: 5
        resources:
          requests:
            cpu: "100m"
            memory: "64Mi"
          limits:
            cpu: "500m"
            memory: "128Mi"

kubectl Commands

kubectl apply -f app-period.yaml
kubectl get deployments delayed-app
kubectl get pods -l app=delayed-app
kubectl logs delayed-app-7w8d6q6b -f

Expected Outcome

The pod should remain in the `Running` state. The increased `periodSeconds` ensures that the probe checks the readiness of the application every 20 seconds, providing more frequent monitoring and faster detection of any issues during initialization.

Conclusion

By understanding the role of startup probes and adjusting parameters like `initialDelaySeconds` and `periodSeconds`, you can effectively resolve startup probe failures caused by application initialization delays. Carefully monitoring pod status and logs allows you to diagnose the root cause and ensure your applications are healthy and responsive within the Kubernetes environment.