Kubernetes Tutorial: Repairing Pending Pods Blocked by Taints and Missing Tolerations

This tutorial walks you through the process of diagnosing and resolving a common Kubernetes issue: a pending pod stuck in a ‘FailedScheduling’ state due to missing tolerations for taints applied to a node. We’ll use YAML manifests and `kubectl` commands to demonstrate the steps.

Understanding Taints and Tolerations

Taints are labels applied to nodes that indicate a node is suitable for certain pods but not others. Pods without the corresponding tolerations will be denied scheduling on those tainted nodes.

Here’s a quick breakdown:

• Taint: A label applied to a node to restrict pod scheduling.
• Toleration: A label added to a pod to allow it to schedule on a node with a matching taint.

Example 1: Initial Problem – Unscheduled Pod

Let’s start with a scenario. We have a node named `worker-node-1` that’s been tainted to only allow pods with the label `app=backend`. A pod named `my-backend-pod` is created without a `tolerations` section in its YAML manifest.

apiVersion: v1
kind: Pod
metadata:
  name: my-backend-pod
  labels:
    app: backend
spec:
  containers:
  - name: backend-container
    image: nginx
    ports:
    - containerPort: 80

Now, let’s try to apply this pod to our cluster:

kubectl apply -f my-backend-pod.yaml

The output will likely show something like this:

Unable to create pod my-backend-pod (FailedScheduling)
...
Reason: FailedScheduling
...
Events:
  Type     Reason                 Age    From                     Message
  ----     ------                 ----   ----                     -------
  Warning  FailedScheduling        0s    kubelet/worker-node-1   No nodes match pod's selector

This indicates the pod is failing to schedule because no nodes satisfy the pod’s selector and the taint on `worker-node-1`. The key is the `Reason: FailedScheduling`.

Example 2: Adding a Toleration to the Pod

To resolve this, we need to add a toleration to the `my-backend-pod` YAML manifest. This tells the pod that it can run on the `worker-node-1` node, despite the taint.

apiVersion: v1
kind: Pod
metadata:
  name: my-backend-pod
  labels:
    app: backend
spec:
  containers:
  - name: backend-container
    image: nginx
    ports:
    - containerPort: 80
  tolerations:
  - key: app
    operator: Equal
    value: backend

Apply the modified pod definition:

kubectl apply -f my-backend-pod.yaml

Check the pod’s status:

kubectl get pod my-backend-pod -o wide

The output should now show the pod in a `Running` state, indicating it’s successfully scheduled on `worker-node-1`. Look for a `NODE` column showing `worker-node-1`.

Example 3: Correcting the Node Taint (If Necessary – Advanced)

Sometimes the issue isn’t the pod but the node’s taint. Let’s imagine the taint on `worker-node-1` was inadvertently set. We need to remove it, or modify it to allow scheduling.

kubectl describe node worker-node-1

Examine the output. Look for the `Taints` section. If it exists and is preventing scheduling, you can remove it with:

kubectl taint node worker-node-1 app=backend:NoSchedule

This removes the taint. If you want to allow scheduling, you can change the effect from `NoSchedule` to `Preferred` or `Equal`, depending on your requirements. (Only do this if you understand the implications of changing the taint effect.)

After removing or modifying the taint, try applying the `my-backend-pod` again. It should schedule successfully.