Kubernetes: Analyzing Node Affinity Scheduling Issues

This tutorial focuses on diagnosing and resolving scheduling issues within a Kubernetes Deployment caused by Node Affinity constraints. We’ll use YAML manifests and `kubectl` commands to identify and correct the problem. The goal is to understand how node affinity affects pod placement and learn to troubleshoot the resulting scheduling conflicts.

Understanding Node Affinity

Node affinity allows you to constrain pods to run on specific nodes based on their labels. This is useful for resource isolation, affinity for specific hardware (if available), or dedicated workloads. Incorrectly configured node affinity can prevent pods from scheduling successfully.

Key concepts:

• Required During Scheduling: Node affinity rules are evaluated during the scheduling process.
• No Guarantee: While node affinity attempts to match pods to nodes, Kubernetes still uses other factors (resource requests, taints, tolerations, etc.) to determine placement.
• Rules: Node affinity comprises two types of rules: requiredDuringSchedulingIgnoredDuringExecution and preferredDuringSchedulingIgnoredDuringExecution. The first enforces the rule, the second suggests it.

Example 1: Simple Node Affinity Conflict

Let’s start with a basic scenario where we create two Deployments, each with a node affinity rule that restricts them to different nodes.

# deployment-a.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: deployment-a
spec:
  replicas: 2
  selector:
    matchLabels:
      app: a
  template:
    metadata:
      labels:
        app: a
    spec:
      containers:
      - name: container-a
        image: busybox
        command: ["/bin/sh", "-c", "while true; do echo 'Container A running' & sleep 5; done"]
      nodeAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          nodeSelectorTerms:
          - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values:
              - node-a

# deployment-b.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: deployment-b
spec:
  replicas: 2
  selector:
    matchLabels:
      app: b
  template:
    metadata:
      labels:
        app: b
    spec:
      containers:
      - name: container-b
        image: busybox
        command: ["/bin/sh", "-c", "while true; do echo 'Container B running' & sleep 5; done"]
      nodeAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          nodeSelectorTerms:
          - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values:
              - node-b

Now, let’s create the nodes:

kubectl apply -f deployment-a.yaml
kubectl apply -f deployment-b.yaml

After applying these manifests, observe the following in your cluster:

kubectl get deployments

You’ll likely see that only one of the deployments is running, or that pods are failing to schedule. This is because `deployment-a` is tied to `node-a` and `deployment-b` is tied to `node-b`. The scheduler cannot satisfy both rules simultaneously.

Example 2: Correcting the Conflict – Preferred Affinity

To allow both deployments to run, we can use preferredDuringSchedulingIgnoredDuringExecution with a weight to express a preference.

# deployment-a-preferred.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: deployment-a
spec:
  replicas: 2
  selector:
    matchLabels:
      app: a
  template:
    metadata:
      labels:
        app: a
    spec:
      containers:
      - name: container-a
        image: busybox
        command: ["/bin/sh", "-c", "while true; do echo 'Container A running' & sleep 5; done"]
      nodeAffinity:
        preferredDuringSchedulingIgnoredDuringExecution:
          nodeSelectorTerms:
          - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values:
              - node-a
            weight: 1000

# deployment-b-preferred.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: deployment-b
spec:
  replicas: 2
  selector:
    matchLabels:
      app: b
  template:
    metadata:
      labels:
        app: b
    spec:
      containers:
      - name: container-b
        image: busybox
        command: ["/bin/sh", "-c", "while true; do echo 'Container B running' & sleep 5; done"]
      nodeAffinity:
        preferredDuringSchedulingIgnoredDuringExecution:
          nodeSelectorTerms:
          - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values:
              - node-b
            weight: 1000

Apply these corrected manifests. The scheduler will now prefer `node-a` for `deployment-a` and `node-b` for `deployment-b`, increasing the chances of both deployments succeeding.

kubectl get deployments
kubectl get pods -o wide

Example 3: Debugging with Events and Describing Pods

If scheduling issues persist, use `kubectl describe pod ` to examine the events related to the pod’s scheduling. This will often reveal the exact reason for the failure, including the node affinity constraints that are preventing the pod from being scheduled.

kubectl describe pod deployment-a-pod-1  #Replace with actual pod name

Look for entries in the `Events` section that indicate node affinity constraints or other scheduling issues. For example, you might see something like “PodFitsHostAddresses” or a message related to the node affinity rules. Examine the `NodeAffinity` section within the pod spec as well.

This tutorial has covered a fundamental aspect of Kubernetes scheduling: Node Affinity. By understanding and correctly configuring these constraints, you can improve the reliability and performance of your deployments.