Default Toleration at Namespace Level
I have an OpenShift cluster where I dedicate some of the nodes to run my workload by tainting these nodes. To run my normal pods on these nodes, I just need to define the tolerations based on the taint keys.
However, the workload is operator-based, and too bad not all the CRD has the tolerations defined. The “brute-force change” on the Deployment or the Statefulset will not take effect in the end as “the big brother” will rectify it based on the definition in its original mind ;)
I need the missing toleration to be inserted after the operator creates the K8s resource and before the resource persists and submits for scheduling. This is what the admission controllers, particularly PodTolerationRestriction can help.
The admission controller PodTolerationRestriction will check if the Pod tolerations conflict with the predefined whitelist, and it is able to define default tolerations at the namespace level with annotation. If the pod doesn’t have the toleration then this default toleration will be applied.
Let’s check it out.
1. Taint the node
Taint all the worker nodes non-schedulable.
kubectl taint nodes {{ .node }} reservedFor=myApp:NoSchedule