Kubernetes Liveness Probe Validation in a Complex Application

This tutorial demonstrates how to diagnose and resolve a failing liveness probe in a Kubernetes application. We’ll focus on understanding the process of identifying the root cause and then adjusting the probe configuration to ensure healthy application behavior. Our goal is to illustrate how a seemingly simple Kubernetes feature can become critical in debugging complex application issues.

Scenario: A Complex Application with Multiple Services

We’ll simulate a scenario where we have a microservice architecture. Specifically, we have an app-frontend service and an app-backend service. The app-frontend depends on the app-backend. The app-backend is more complex and susceptible to simulated issues we’ll introduce to test the liveness probe validation process.

Example 1: Initial Setup and a Working Liveness Probe

First, let’s create the initial YAML manifests for both services. This assumes a basic understanding of Kubernetes concepts.

app-backend.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: app-backend
  labels:
    app: app-backend
spec:
  replicas: 2
  selector:
    matchLabels:
      app: app-backend
  template:
    metadata:
      labels:
        app: app-backend
    spec:
      containers:
      - name: app-backend
        image: busybox:latest
        ports:
        - containerPort: 8080
        command: ["/bin/sh", "-c", "while true; do echo 'App Backend Running' & sleep 5; done"]
        livenessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 15
          periodSeconds: 10
        resources:
          requests:
            cpu: "100m"
            memory: "128Mi"
          limits:
            cpu: "200m"
            memory: "256Mi"

app-frontend.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: app-frontend
  labels:
    app: app-frontend
spec:
  replicas: 2
  selector:
    matchLabels:
      app: app-frontend
  template:
    metadata:
      labels:
        app: app-frontend
    spec:
      containers:
      - name: app-frontend
        image: busybox:latest
        ports:
        - containerPort: 80
        command: ["/bin/sh", "-c", "while true; do echo 'App Frontend Running' & sleep 5; done"]
        readinessProbe:
          httpGet:
            path: /health
            port: 80
          initialDelaySeconds: 5
          periodSeconds: 10
        livenessProbe:
          httpGet:
            path: /health
            port: 8080
        resources:
          requests:
            cpu: "50m"
            memory: "64Mi"
          limits:
            cpu: "100m"
            memory: "128Mi"

Now, apply these manifests:

kubectl apply -f app-backend.yaml app-frontend.yaml

Verify the deployments:

kubectl get deployments app-backend app-frontend

Check the logs of the backend service. You should see the “App Backend Running” message repeatedly. This confirms the liveness probe is successfully passing.

kubectl logs -f app-backend-7b6d4b9c6-b5k9q -c app-backend

Example 2: Introducing a Simulated Failure – Backend Service Crash

Let’s simulate a failure in the app-backend service. We’ll intentionally introduce a loop that quickly exhausts resources, causing the container to crash. We’ll observe the liveness probe behavior.

Modify the app-backend.yaml file as follows (specifically changing the command section):

    command: ["/bin/sh", "-c", "while true; do echo 'App Backend Running' & sleep 0.1; done"]

Apply the changes:

kubectl apply -f app-backend.yaml

Check the deployment status:

kubectl get deployments app-backend

Observe the logs of the app-backend service. You should see the container repeatedly crashing. The liveness probe will immediately start failing. The `app-frontend` deployment will remain healthy, as its liveness probe depends on the app-backend.

kubectl logs -f app-backend-7b6d4b9c6-b5k9q -c app-backend

Check the events for the deployment:

kubectl get events --sort-by=.lastTimestamp

Look for events related to the app-backend deployment indicating liveness probe failures. You will likely see messages like “Liveness probe failed” repeated.

Example 3: Correcting the Liveness Probe – Resource Limit Adjustment

To resolve the liveness probe failure, we need to adjust the resources allocated to the app-backend. The initial command in Example 2 was extremely aggressive, overwhelming the container.

Modify the app-backend.yaml file, specifically the resources section, to the following:

    resources:
      requests:
        cpu: "100m"
        memory: "128Mi"
      limits:
        cpu: "500m"
        memory: "512Mi"

Apply the changes:

kubectl apply -f app-backend.yaml

Check the deployment status again:

kubectl get deployments app-backend

Verify that the liveness probe is passing now. The container will run without crashing. The logs will show “App Backend Running” and the liveness probe will report success. Examine the events again. They should now indicate that the liveness probe is passing.

kubectl get events --sort-by=.lastTimestamp

Finally, check the liveness probe status:

kubectl describe probe app-backend

This demonstrates how a failing liveness probe can be identified and corrected by adjusting resource constraints, showcasing the core functionality of Kubernetes’ health checking mechanisms.