Default Toleration at Namespace Level

I have an OpenShift cluster where I dedicate some of the nodes to run my workload by tainting these nodes. To run my normal pods on these nodes, I just need to define the tolerations based on the taint keys.

However, the workload is operator-based, and too bad not all the CRD has the tolerations defined. The “brute-force change” on the Deployment or the Statefulset will not take effect in the end as “the big brother” will rectify it based on the definition in its original mind ;)

I need the missing toleration to be inserted after the operator creates the K8s resource and before the resource persists and submits for scheduling. This is what the admission controllers, particularly PodTolerationRestriction can help.

The admission controller PodTolerationRestriction will check if the Pod tolerations conflict with the predefined whitelist, and it is able to define default tolerations at the namespace level with annotation. If the pod doesn’t have the toleration then this default toleration will be applied.

Let’s check it out.

1. Taint the node

Taint all the worker nodes non-schedulable.

kubectl taint nodes {{ .node }} reservedFor=myApp:NoSchedule

2. Create a test app in a test namespace

Create a namespace “tolerant”. Deploy a test app in this namespace as shown below,

apiVersion: apps/v1
kind: Deployment
metadata:
  name: app
  namespace: tolerant
  labels:
    app: app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: app
  template:
    metadata:
      labels:
        app: app
    spec:
      containers:
      - name: app
        image: alpine
        command:
          - sh
          - -c
          - while true; do sleep 10; done

After applying it, the pod is in pending mode. Describe it, we see the following,

Warning  FailedScheduling  10s (x3 over 27s)  default-scheduler  0/4 nodes are available: 1 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate, 3 node(s) had taint {reservedFor: myApp}, that the pod didn't tolerate.