Custom Grafana Dashboard for User Workload in OpenShift
I have written a medium blog at the beginning of 2020, Grafana Dashboard in OCP4.2, where we install our own grafana to display some custom dashboards. With the formal user workload monitoring support (since OCP4.6), it’s time to re-examine the Grafana dashboard with the new monitoring stack in OCP.
Install Grafana for User Workload Monitoring
The Grafana comes with OCP is targeted for cluster monitoring and its read-only. We have to install our own Grafana for custom dashboards.
To enjoy the latest Grafana 8, let's install the Helm chart from grafana.com.
Add the helm repo,
helm repo add grafana https://grafana.github.io/helm-charts
Create the namespace,
kubectl create ns grafana
Prepare the following
Install the chart,
helm -n grafana upgrade my-grafana grafana/grafana -f values.yaml
You will notice that the Pod is pending from the replica set simply because the SCC constraint is not met, fix it by,
oc adm policy add-scc-to-user anyuid -z my-grafana -n grafana
The service account is using the chart’s release name.
Delete the replica set and let the deployment resource regenerate it, now you should have the Grafana running.
Launch the route URL, find the admin username and password from the secret of
my-grafana login and now we have the Grafana 8.
Configure Prometheus Data Source
In OCP, both cluster Prometheus and user workload Prometheus are not exposed to the cluster. You can not access the Prometheus directly as they are bound to 127.0.0.1 only. It seems like user workload Prometheus is not made available even through sidecars.
After OCP 4.6, a better approach is to use the Thanos Querier as the Prometheus data source, wherein turn the queries will goto cluster Prometheus or user Prometheus respectively.
We have to let grafana be able to access the Thanos querier. Add the cluster role of cluster-monitoring-view to the grafana service account,
oc adm policy add-cluster-role-to-user cluster-monitoring-view -z my-grafana -n grafana
Get its token and copy/note down the token.
oc sa get-token my-grafana -n grafana
Now go back to the Grafana web console, add the Prometheus data source. Take the Thanos querier service from the namespace of openshift-monitoring.
Skip the TLS Verify.
Add a custom HTTP header,
Authorizationwith the value of
The token after “Bearer” is what we copied.
Save and test the data source to make sure it is working.
Now we can create our own dashboard with the cluster metrics and user workload metrics.