Create alerts for OpenShift user workload

Starting from OpenShift 4.6, user workload monitoring is formally supported by introducing a second Prometheus operator instance in a new namespace called openshift-user-workload-monitoring. This paper demonstrates how a user workload can be monitored and the alerts can be created.

Turn on user workload monitoring

Create or update the following configMap in openshift-monitoring

apiVersion: v1
kind: ConfigMap
metadata:
  name: cluster-monitoring-config
  namespace: openshift-monitoring
data:
  config.yaml: |
    enableUserWorkload: true

With the enableUserWorkload key set as true, the 2nd Prometheus operator will be installed, which will create a Prometheus and Thanos Ruler as shown as below,

oc -n openshift-user-workload-monitoring get pods
NAME                                   READY   STATUS    RESTARTS   AGE
prometheus-operator-7bd67b9d5d-znr8r   2/2     Running   0          19h
prometheus-user-workload-0             5/5     Running   1          19h
prometheus-user-workload-1             5/5     Running   1          19h
thanos-ruler-user-workload-0           3/3     Running   0          19h
thanos-ruler-user-workload-1           3/3     Running   0          19h