I need to run some monitoring in a restricted environment. It is restricted in the sense of first its air-gapped, no internet connection is there. Secondly, the environment is kind of locked, no extra dependency modules can be installed.

The lightweight Kubernetes distribution, K3s, with its self-contained dependencies serves the requirement perfectly. This paper will show how I set up the Prometheus in the above-mentioned restricted environment.

Install K3s in the airgap server

The airgap setup for K3s is well documented. The following is a quick command list of the packages and scripts that need to be downloaded on an internet-facing server.

curl -LO https://github.com/k3s-io/k3s/releases/download/v1.21.1%2Bk3s1/k3s-airgap-images-amd64.tar.gzcurl…

Recently I need to test some J2EE integration work on IBM MQ. Since it is for testing purposes I run it as a container instead of the full-blown VM.

Preparing a custom QMSC file

Other than the default MQ settings, I need to use my custom queue definition and security controls. In the MQ container, this is achieved by defining a custom MQSC file and put it into the specific /etc/mqm directory.

Create the following file 80-my-test.mqsc

define listener(LISTENER) trptype(tcp) control(qmgr) port(1414) replace
start listener(LISTENER)define authinfo(my.authinfo) authtype(idpwos) chckclnt(reqdadm) chcklocl(optional) adoptctx(yes) replace
alter qmgr connauth(my.authinfo)
refresh security(*) type(connauth)def chl(SYSTEM.ADMIN.SVRCONN) chltype(SVRCONN) replaceset chlauth('*') type(addressmap)…

Starting from OpenShift 4.6, user workload monitoring is formally supported by introducing a second Prometheus operator instance in a new namespace called openshift-user-workload-monitoring. This paper demonstrates how a user workload can be monitored and the alerts can be created.

Turn on user workload monitoring

Create or update the following configMap in openshift-monitoring

apiVersion: v1
kind: ConfigMap
metadata:
  name: cluster-monitoring-config
  namespace: openshift-monitoring
data:
  config.yaml: |
    enableUserWorkload: true

With the enableUserWorkload key set as true, the 2nd Prometheus operator will be installed, which will create a Prometheus and Thanos Ruler as shown as below,

oc -n openshift-user-workload-monitoring get pods
NAME…

Let's explore how we can integrate an OpenID Connect (OIDC) implementation, keycloak, as an identity provider for OpenShift, other than the common one such as HTTPasswd, LDAP.

Setup Keycloak on OpenShift

Install the Keycloak operator from the OperatorHub, create a keycloak instance in the namespace of keycloak. We can access the admin web interface once the pods are running. Get the admin user and its password from the corresponding Secret object in the namespace.

Create our own realm, myrealm, first.

Secondly, create the client named idp-4-ocp. In the settings tab, select the “Access Type” as “confidential”. Set the “Validation Redirection URIs” as “https://*” for…

Airgap installation is always a challenging thing for OpenShift. By setting up a mirror registry and applying ImageContentSourcePolicy CRD to the cluster, we can instruct the OCI container engine to retrieve the source image from its mirrored image hosted in the mirror registry. This solves the airgap images for the cluster and the apps.

There is still a 3rd type of image for an airgap environment to tackle, that is the Operator related images. This paper documents the Operator based installation in an air-gapped environment, the steps, and the hiccups, and how it is being resolved.

OpenShift manages operators through…

With the GA release of volume snapshots, the CSI volume snapshot for stateful application data backup and restore is more mature. This paper explores how we can use the standard Kubernetes snapshot resources to backup and restore the data located on the PV.

Let's use the IBM event streams as the test target. It is running a statefulset K8s resource of Kafka based on the Strimizi Operator. Assume we have 3 replicas of the statefulset and the data are saved in the PVC named as data-es-kafka-0, data-es-kafka-1, data-es-kafka-2 respectively. The PVCs are provided by the Rook Ceph.

Volume snapshot class

Just as PVC…

Kafka provides a rich set of command-line tools to manage the topics and clusters. The default settings of the latest IBM Event Streams (Strimzi Operator based) gives some challenges to run these tools.

Broker Listeners

When a Kafka resource is deployed as an operator, a strong security configuration is applied normally. Take a look at the following excerpt of the cluster listener settings

listeners:
  external:
    authentication:
      type: scram-sha-512
    type: route
  tls:
    authentication:
      type: tls

The brokers will be listening on

1. Port 9094 for external connections with SRAM-SHA-512 authentication.
2. Port 9093 for internal communication, where mTLS are used. The m here (mutual) means…

The Etcd is the heart of the Kubernetes. With the operator model prevailing, the Etcd is no longer limited to the usage of the Kubernetes core cluster engine only.

Following is a screen capture of the major Etcd metrics on my OpenShift cluster when an operator based solution framework is deployed. You can see both DB size and the Memory are increased 3 to 4 times on the plain OCP platform.