Kubernetes Tutorial: Repairing Pending Pods – Taints & Tolerations

This tutorial guides you through troubleshooting and resolving Kubernetes Pods stuck in a ‘Pending’ state due to misconfigured Taints and Tolerations. We’ll walk through the process step-by-step using YAML manifests and `kubectl` commands.

Understanding Taints and Tolerations

Taints are labels attached to nodes that restrict which Pods can be scheduled onto them. Tolerations are similar labels applied to Pods that allow them to be scheduled onto nodes with matching Taints. When a Pod lacks a Toleration required by a Node’s Taint, the Pod remains in the Pending state.

Here’s a simple breakdown:

• Taint: A “no entry” rule for a node.
• Toleration: A “permission” granted to a Pod.

Example 1: Initial Problem – A Tainted Node

Let’s create a scenario where a node is intentionally tainted, and then a Pod attempts to be scheduled on it without the necessary Toleration.

Step 1: Create a Tainted Node

apiVersion: v1
kind: Node
metadata:
  name: tainted-node
  labels:
    kubernetes.io/instance: tainted-node
spec:
  taints:
  -   apiVersion: v1
      kind: Taint
      name: node-role.kubernetes.io/worker
      effect: NoSchedule

This manifests creates a node named `tainted-node` with a Taint applied. The Taint’s name is `node-role.kubernetes.io/worker` and its effect is `NoSchedule`. This means only Pods without a matching Toleration can be scheduled on this node.

Step 2: Create a Pod with a Missing Toleration

apiVersion: v1
kind: Pod
metadata:
  name: worker-pod-no-toleration
  labels:
    app: worker
spec:
  containers:
  - name: worker-container
    image: busybox
    command: ["sleep", "3600"]
  restartPolicy: Always

This creates a simple Pod named `worker-pod-no-toleration` which attempts to run a busybox container. It doesn’t include a Toleration.

Step 3: Observe the Pending State

kubectl get pods

The output should show the `worker-pod-no-toleration` pod in the Pending state. Check the events for details:

kubectl describe events --namespace default

You’ll see an event stating that the Pod failed to schedule due to a node-level Taint it doesn’t tolerate.

Example 2: Adding the Necessary Toleration

Now, let’s add a Toleration to the Pod to allow it to schedule on the tainted node.

Step 1: Update the Pod Spec

apiVersion: v1
kind: Pod
metadata:
  name: worker-pod-with-toleration
  labels:
    app: worker
spec:
  containers:
  - name: worker-container
    image: busybox
    command: ["sleep", "3600"]
  restartPolicy: Always
  tolerations:
  -   apiVersion: v1
      kind: Toleration
      operator: "Equal"
      effect: "NoSchedule"
      tolerationSeconds: 0
      key: "node-role.kubernetes.io/worker"

We’ve added a Toleration to the Pod’s specification. This Toleration matches the Taint’s name and effect. The `tolerationSeconds` is set to 0, meaning the toleration always applies.

Step 2: Attempt to Schedule the Pod

kubectl get pods

The `worker-pod-with-toleration` pod should now be in the Running state.

Example 3: Correcting the Taint (More Advanced)

Sometimes, Taints are applied unintentionally. Let’s demonstrate how to remove or modify a Taint.

Step 1: Remove the Taint from the Node

kubectl delete node tainted-node

This removes the Taint from the `tainted-node`. Without the Taint, any Pod can be scheduled on that node.

Step 2: Verify Pod Scheduling

kubectl get pods

The `worker-pod-with-toleration` (or a new Pod) should now be scheduled on the `tainted-node` without any issues, as the Taint is no longer present.

Conclusion

This tutorial demonstrated how to troubleshoot and resolve a common Kubernetes issue: Pending Pods blocked by Taints and missing Tolerations. By understanding the interplay between Taints and Tolerations, and utilizing `kubectl` commands to examine Pod events and modify resource specifications, you can effectively manage your Kubernetes deployments.