Kubernetes Tutorial: Repairing Pending Pods Blocked by Taints and Missing Tolerations

This tutorial guides you through resolving a common Kubernetes issue: a pending pod stuck in a waiting state because it lacks the necessary tolerations to satisfy taints applied to a specific node or namespace. We will walk through three progressively complex scenarios.

Understanding Taints and Tolerations

Before we begin, let’s recap the key concepts:

• Taints: Attributes applied to nodes (or namespaces) that restrict which pods can be scheduled onto them. They essentially say, “Only pods with this label will be allowed here.”
• Tolerations: Attributes that pods have to override taints. They say, “I don’t care about that taint; I can still be scheduled here.”

A pod will only be scheduled on a node if it has a toleration that matches the taints on that node.

Example 1: Initial Problem – A Simple Taint and Pending Pod

We’ll start with a basic setup to demonstrate the problem.

Step 1: Create a Tainted Node

kubectl create node my-tainted-node --taints=name=gpu-node,kubernetes.io/arch=amd64

This command creates a node named `my-tainted-node` and applies a taint: pods will be denied scheduling on this node unless they have a toleration for `name=gpu-node` and `kubernetes.io/arch=amd64`. This creates a resource named `nodes/my-tainted-node`

Step 2: Deploy a Pod with No Tolerations

apiVersion: v1
kind: Pod
metadata:
  name: gpu-pod
  labels:
    app: my-app
spec:
  containers:
  - name: my-container
    image: busybox
    command: ["sleep", "3600"]
  nodeName: my-tainted-node # Explicitly schedules on the tainted node

This YAML defines a pod named `gpu-pod` that tries to be scheduled on `my-tainted-node`. It doesn’t have any tolerations, so it will remain in a Pending state.

Step 3: Verify the Pending State

kubectl get pods

NAME READY STATUS REASONS
gpu-pod 0/1 Pending FailedScheduling

The output shows `gpu-pod` in a `Pending` state with the reason `FailedScheduling`. Kubernetes cannot schedule it because it lacks the required tolerations.

Example 2: Adding a Toleration – Correcting the Problem

Now, let’s add a toleration to the pod to allow it to run on the tainted node.

Step 1: Add the Toleration to the Pod

apiVersion: v1
kind: Pod
metadata:
  name: gpu-pod
  labels:
    app: my-app
spec:
  containers:
  - name: my-container
    image: busybox
    command: ["sleep", "3600"]
  nodeName: my-tainted-node
  tolerations:
  - key: name
    operator: Equal
    value: gpu-node
    effect: NoSchedule

This YAML adds a toleration to the `gpu-pod`. The `key`, `operator`, `value`, and `effect` define how this toleration matches the taint. The `effect: NoSchedule` means the pod will be scheduled despite the taint.

Step 2: Verify the Pod’s Status

kubectl get pods

NAME READY STATUS REASONS
gpu-pod 1/1 Running

The pod is now in a `Running` state. The critical `FailedScheduling` reason is gone.

Example 3: Taint Propagation Across Namespaces – An Advanced Scenario

This example introduces a namespace and demonstrates that taints propagate to namespaces. We’ll create a second node and then try to schedule a pod to a namespace that inherits the taint.

Step 1: Create a Namespace with the Taint

kubectl create namespace gpu-namespace
kubectl taint namespace gpu-namespace app=gpu-node:NoSchedule

This creates a namespace named `gpu-namespace` and applies a taint to it. Any pod scheduled into this namespace will be subject to this taint.

Step 2: Deploy a Pod in the New Namespace

apiVersion: v1
kind: Pod
metadata:
  name: gpu-pod-ns
  namespace: gpu-namespace
  labels:
    app: my-app
spec:
  containers:
  - name: my-container
    image: busybox
    command: ["sleep", "3600"]
  nodeName: my-tainted-node

This YAML defines a pod in the `gpu-namespace`. The pod is still subject to the taint applied to the namespace.

Step 3: Verify Pod Status

kubectl get pods -n gpu-namespace

NAME READY STATUS REASONS
gpu-pod-ns 0/1 Pending FailedScheduling

The pod remains Pending. The taint is still being enforced at the namespace level. You’ll need to add a toleration specifically for the namespace to allow scheduling.

Step 4: Add Namespace Toleration

apiVersion: v1
kind: Pod
metadata:
  name: gpu-pod-ns
  namespace: gpu-namespace
  labels:
    app: my-app
spec:
  containers:
  - name: my-container
    image: busybox
    command: ["sleep", "3600"]
  nodeName: my-tainted-node
  tolerations:
  - key: name
    operator: Equal
    value: gpu-node
    effect: NoSchedule
    range: PodScope

The `range: PodScope` ensures this toleration applies to pods within this namespace.

Step 5: Verify Pod Status (Again)

kubectl get pods -n gpu-namespace

NAME READY STATUS REASONS
gpu-pod-ns 1/1 Running

The pod is now Running. The addition of the namespace-scoped toleration resolved the issue.