I have written a medium blog at the beginning of 2020, Grafana Dashboard in OCP4.2, where we install our own grafana to display some custom dashboards. With the formal user workload monitoring support (since OCP4.6), it’s time to re-examine the Grafana dashboard with the new monitoring stack in OCP.

The Grafana comes with OCP is targeted for cluster monitoring and its read-only. We have to install our own Grafana for custom dashboards.

To enjoy the latest Grafana 8, let's install the Helm chart from grafana.com.

Add the helm repo,

helm repo add grafana https://grafana.github.io/helm-charts

Create the namespace,

kubectl create ns…

When people focusing more on the security of containers, distroless based images are frequently used to reduce the attack surface. In these images, the package manager, the non-dependent modules or libraries, even the shells are stripped off, only the app and its required dependencies are kept. For the statically linked executable, produced by golang for example, we can even use “scratch” as the base.

The potential exploit of vulnerability is therefore greatly reduced. But, on the other hand, it is difficult to troubleshoot the application if even the shell is not available, leaving only the logs from the app.


I am tracking some EOD job status to check if it starts late than the scheduled time, or overruns. Created as a Prometheus client program with Golang, the data are collected as a Prometheus gauge metric to be displayed in the Grafana dashboard using the table format.

Worked and tested perfectly on my laptop.

However, I had the typical problem that we commonly encountered. Yes, it worked on my laptop ;) It doesn’t work when I deployed into Kubernetes running as a container!

After some troubleshooting, I realized it's the default timezone in the container that causes the issue.


I have a request to perform some Google API search for some keywords and save the result (the URLs) into text files. It's a good chance to use Go’s concurrency pattern to assembly the pipeline, therefore, parallelizing the IO and CPU.

Given a keyword, say a category, the following function calls Google search API using go-resty, then picks up the respective Links from the result using the gjson library.

func SearchGoogle(category string, count int) ([]SearchResult, error) {
results := []SearchResult{}
// ... skip some initial assignment
for i := 0; i < pages; i++ {
resp, err := client.R()…

I need to mine some business metrics from a DB2 database and present it in the Grafana dashboard. A Prometheus scraping target is to be developed.

I published a Medium paper about 3 years ago to run DB2 queries in Golang by using the DB2 ODBC/CLI driver. Following that, let's create a container image as a Prometheus scraping target and run it in Kubernetes.

The app as a scraping target

Some Golang code excerpts to describe how the data collection works,

The DB and metric struct

type MetricConfig struct {
Name string `yaml:"name"`
Desc string `yaml:"desc"`
Sql string `yaml:"sql"`
Frequency string `yaml:"frequency"`
type DBMetricsConfig struct…

I need to run some monitoring in a restricted environment. It is restricted in the sense of first its air-gapped, no internet connection is there. Secondly, the environment is kind of locked, no extra dependency modules can be installed.

The lightweight Kubernetes distribution, K3s, with its self-contained dependencies serves the requirement perfectly. This paper will show how I set up the Prometheus in the above-mentioned restricted environment.

The airgap setup for K3s is well documented. The following is a quick command list of the packages and scripts that need to be downloaded on an internet-facing server.

curl -LO https://github.com/k3s-io/k3s/releases/download/v1.21.1%2Bk3s1/k3s-airgap-images-amd64.tar.gzcurl…


Recently I need to test some J2EE integration work on IBM MQ. Since it is for testing purposes I run it as a container instead of the full-blown VM.

Other than the default MQ settings, I need to use my custom queue definition and security controls. In the MQ container, this is achieved by defining a custom MQSC file and put it into the specific /etc/mqm directory.

Create the following file 80-my-test.mqsc

define listener(LISTENER) trptype(tcp) control(qmgr) port(1414) replace
start listener(LISTENER)
define authinfo(my.authinfo) authtype(idpwos) chckclnt(reqdadm) chcklocl(optional) adoptctx(yes) replace
alter qmgr connauth(my.authinfo)
refresh security(*) type(connauth)
def chl(SYSTEM.ADMIN.SVRCONN) chltype(SVRCONN) replaceset chlauth('*') type(addressmap)…

Starting from OpenShift 4.6, user workload monitoring is formally supported by introducing a second Prometheus operator instance in a new namespace called openshift-user-workload-monitoring. This paper demonstrates how a user workload can be monitored and the alerts can be created.

Turn on user workload monitoring

Create or update the following configMap in openshift-monitoring

apiVersion: v1
kind: ConfigMap
name: cluster-monitoring-config
namespace: openshift-monitoring
config.yaml: |
enableUserWorkload: true

With the enableUserWorkload key set as true, the 2nd Prometheus operator will be installed, which will create a Prometheus and Thanos Ruler as shown as below,

oc -n openshift-user-workload-monitoring get pods

Let's explore how we can integrate an OpenID Connect (OIDC) implementation, keycloak, as an identity provider for OpenShift, other than the common one such as HTTPasswd, LDAP.

Install the Keycloak operator from the OperatorHub, create a keycloak instance in the namespace of keycloak. We can access the admin web interface once the pods are running. Get the admin user and its password from the corresponding Secret object in the namespace.

Create our own realm, myrealm, first.

Secondly, create the client named idp-4-ocp. In the settings tab, select the “Access Type” as “confidential”. Set the “Validation Redirection URIs” as “https://*” for…

Airgap installation is always a challenging thing for OpenShift. By setting up a mirror registry and applying ImageContentSourcePolicy CRD to the cluster, we can instruct the OCI container engine to retrieve the source image from its mirrored image hosted in the mirror registry. This solves the airgap images for the cluster and the apps.

There is still a 3rd type of image for an airgap environment to tackle, that is the Operator related images. This paper documents the Operator based installation in an air-gapped environment, the steps, and the hiccups, and how it is being resolved.

OpenShift manages operators through…

Zhimin Wen

Cloud explorer

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store