Monitor IBM MQ with Instana

This paper explores how to configure Instana and MQ to monitor the IBM MQ running on OpenShift.

Instana sensor connects to the detected MQ to get the monitoring metrics. Typically the connection need to be authenticated againt the OS. However, when MQ is running on container, the default OS based authentication is no longer available. To authenticate the connection and further authorize it with the right permission, we will use mutual TLS, certificate based authentication for the Instana sensor.

The Instana agent has been deployed as instana-agent helm chart. The agent is running on each nodes of the cluster as Daemonset. If the MQ is running on any of the node, the Instana agent will discover it and start to monitor it based on a set of predefined configuration. The configuration is created as a configMap. The following is a sample configMap for the MQ monitoring,

kind: ConfigMap
apiVersion: v1
metadata:
name: instana-agent
namespace: instana-agent
data:
cluster_name: ocp-dev49
configuration.yaml: |
com.instana.plugin.ibmmq:
enabled: true
poll_rate: 5
queueManagers:
qm1:
channel: INSTANA
#username:
#password:
queuesRegex: '.*'
availabilityZone: 'IBM MQ'
keystore: /certs/mqplugin.jks
keystorePassword: 'password'
cipherSuite: 'TLS_RSA_WITH_AES_256_CBC_SHA256'
poll_rate: 5

The MQ monitoring is defined through the com.instana.plugin.ibmmq in the configuration.yaml.

For each queue manager to be monitored, set its properties under the queueManagers. Because of this setting, the queue manager’s name have to be unique. For the above example, the qm1 queue manager is configured as below,

  • Connect through the MQ channel named as INSTANA. (case sensitive)
  • The username and password are not used as we are going to use the certificate based authentication.
  • The MQ port and host is not required here as the Instana agent will discover the local MQ automatically.

The certificate is important as its not only encrypt the traffic but also is used as an authenticate mechanism for the MQ.

Mutual TLS is used. The Instana sensor must be able to validate the cert from the MQ, meanwhile, the queue manager must be able to validate the cert from the sensor also. Therefore we need to use the same CA to create the certificates for the queue manager and the Instana sensor.

The JKS keystore will store the Instana sensor’s certificate and key pair. It also store the CA cert for the validation of the queue manager.

Assume we have created the cert and key pair signed by the same CA with the CN name as CN=mqplugin. Perform the following to create the JKS format keystore,

openssl pkcs12 -export -in certs/mqplugin.pem -inkey certs/mqplugin-key.pem -certfile certs/mqplugin.pem -out mqplugin.p12  -passout pass:password keytool -importkeystore -srckeystore mqplugin.p12 -srcstoretype pkcs12 -destkeystore mqplugin.jks -deststoretype JKS -srcstorepass password -deststorepass password

Import the CA cert into the Keystore to make the validation chain complete.

keytool -importcert -file certs/myca.pem -keystore mqplugin.jks -storepass password -alias mqca -noprompt

We have the mqplugin.jks ready. How we make it available for the Instana agent? The answer is to create the JKS as configMap and mount it to the Pod.

oc -n instana-agent create cm mq-plugin-jks-cm --from-file=mqplugin.jks=mqplugin.jks

As the Daemonset is already created by the Helm, we need to patch it to mount the configMap. Lets use kustomize to achieve this.

mkdir kustomize
cd kustomize
oc -n instana-agent get ds instana-agent -o yaml > instana-agent.ds.yaml

Create the following Kustomize file to patch the Daemonset to mount the configMap.

resources:
- instana-agent.ds.yaml
patchesJSON6902:
- target:
group: apps
version: v1
kind: DaemonSet
name: instana-agent
patch: |-
- op: add
path: /spec/template/spec/volumes/-
value:
name: mq-plugin-jks
configMap:
defaultMode: 420
name: mq-plugin-jks-cm
- op: add
path: /spec/template/spec/containers/0/volumeMounts/-
value:
name: mq-plugin-jks
mountPath: /cert/mqplugin.jks
subPath: mqplugin.jks

Generate the patched YAML and apply it.

cd kustomize
kustomize build > ds.patched.yaml
kubectl apply -f ds.patched.yaml

Watch the pods are restarted and ready. The MQ sensor configuration is completed.

The MQ queque manager is created on OpenShift by using MQ operator. The CR is defined as below,

apiVersion: mq.ibm.com/v1beta1
kind: QueueManager
metadata:
name: my-mq
spec:
license:
accept: true
license: L-RJON-BZFQU2
queueManager:
name: qm1
resources:
limits:
cpu: 500m
requests:
cpu: 500m
storage:
queueManager:
type: persistent-claim
defaultClass: rook-ceph-block
availability:
type: SingleInstance
mqsc:
- configMap:
name: cm-init-mqsc
items:
- init.mqsc
ini:
- configMap:
name: cm-qm-ini-ext-user
items:
- qm.ini
template:
pod:
containers:
- name: qmgr
version: 9.2.3.0-r1
web:
enabled: true
pki:
keys:
- name: mq
secret:
secretName: qm1-tls-secret
items:
- tls.key
- tls.crt
trust:
- name: ca
secret:
secretName: ca-crt-secret
items:
- ca.crt

The pki/keys defines the K8s secret of certificate and key for the queue manager. To enable mutual TLS, the k8s secret of CA is added in pki/trust for the queque manager to validate the cert sent from the Instana sensor.

The ini block disable the look up of OS users in the container. The corresponding configMap is defined as below,

apiVersion: v1
kind: ConfigMap
metadata:
name: cm-qm-ini-ext-user
data:
qm.ini: |-
Service:
Name=AuthorizationService
EntryPoints=14
SecurityPolicy=UserExternal

The mqsc is the MQSC script to configure the queue manager at its first startup. The corresponding configMap is defined as below,

apiVersion: v1
kind: ConfigMap
metadata:
name: cm-init-mqsc
data:
init.mqsc: |-
define channel(instana) chltype(SVRCONN) trptype(TCP) sslcauth(REQUIRED) sslciph('ANY_TLS12_OR_HIGHER')
alter authinfo(SYSTEM.DEFAULT.AUTHINFO.IDPWOS) authtype(IDPWOS) chckclnt(OPTIONAL)
set chlauth(instana) type(SSLPEERMAP) sslpeer('CN=mqplugin') usersrc(MAP) mcauser('mqplugin') action(add)
set authrec principal('mqplugin') objtype(qmgr) authadd(all)
set authrec profile('**') principal('mqplugin') objtype(queue) authadd(all)
set authrec profile('**') principal('mqplugin') objtype(listener) authadd(all)
set authrec profile('**') principal('mqplugin') objtype(topic) authadd(all)

We create a channel, instana, for the instana sensor to connect. It’s noticed if the value is not single quoted, the MQSC will automatically converted to uppercase.

The sslcauth(REQUIRED) set the mutual TLS. The chlauth maps the certificate CN=mqplugin to the MCA user of mqplugin . Then for the MCA user ‘mqplugin’, we assign the proper permission for the queue manager, queues, listener, and topics. The wildcard “**” enables all the objects for that type. Tightening of these permissions can be done once the exact requirements document is published.

Apply the CR and watch the MQ started properly. The Instana sensor will detect the MQ and start to monitor it

Goto Infrastructure, filter by the queue manager’s name, qm1. Click on the found icon in the dashboard, open the queue manager’s dashboard.

Check the Instana MQ sensor’s log

Find which nodes the MQ is running on, then on the same node, find the instana agent pod, run the following to get the MQ sensor related log

oc logs instana-agent-gm28q -c instana-agent | grep ibmmq

Check the MQ log

Exec into the MQ pod, check the detailed MQ log under,

/var/mqm/qmgrs/qm1/errors/AMQERR0[123].LOG

Cloud explorer