Custom Grafana Dashboard for User Workload in OpenShift

I have written a medium blog at the beginning of 2020, Grafana Dashboard in OCP4.2, where we install our own grafana to display some custom dashboards. With the formal user workload monitoring support (since OCP4.6), it’s time to re-examine the Grafana dashboard with the new monitoring stack in OCP.

The Grafana comes with OCP is targeted for cluster monitoring and its read-only. We have to install our own Grafana for custom dashboards.

To enjoy the latest Grafana 8, let's install the Helm chart from grafana.com.

Add the helm repo,

helm repo add grafana https://grafana.github.io/helm-charts

Create the namespace,

kubectl create ns grafana

Prepare the following values.yamlfile.

ingress:
enabled: true
hosts:
- my-grafana.apps.dev-ocp48.ibmcloud.io.cpak
persistence:
type: pvc
enabled: true

Install the chart,

helm -n grafana upgrade my-grafana grafana/grafana -f values.yaml

You will notice that the Pod is pending from the replica set simply because the SCC constraint is not met, fix it by,

oc adm policy add-scc-to-user anyuid -z my-grafana -n grafana

The service account is using the chart’s release name.

Delete the replica set and let the deployment resource regenerate it, now you should have the Grafana running.

Launch the route URL, find the admin username and password from the secret of my-grafana login and now we have the Grafana 8.

In OCP, both cluster Prometheus and user workload Prometheus are not exposed to the cluster. You can not access the Prometheus directly as they are bound to 127.0.0.1 only. It seems like user workload Prometheus is not made available even through sidecars.

After OCP 4.6, a better approach is to use the Thanos Querier as the Prometheus data source, wherein turn the queries will goto cluster Prometheus or user Prometheus respectively.

We have to let grafana be able to access the Thanos querier. Add the cluster role of cluster-monitoring-view to the grafana service account,

oc adm policy add-cluster-role-to-user cluster-monitoring-view -z my-grafana -n grafana

Get its token and copy/note down the token.

oc sa get-token my-grafana -n grafana
eyJhbGciOiJSUzI1NiIsImtp....SKIPPED

Now go back to the Grafana web console, add the Prometheus data source. Take the Thanos querier service from the namespace of openshift-monitoring.

URL: https://thanos-querier.openshift-monitoring.svc:9091

Skip the TLS Verify.

Add a custom HTTP header, Authorizationwith the value of

Bearer eyJhbGciOiJSUzI1NiIsImtp....SKIPPED

The token after “Bearer” is what we copied.

Save and test the data source to make sure it is working.

Now we can create our own dashboard with the cluster metrics and user workload metrics.

Cloud explorer