View OpenShift Router’s Log

To troubleshoot the Router of the OpenShift, the logs are essential, especially the HAProxy logs on how the traffic is handled and forwarded. However, the HAProxy pumping the log to a rsyslog server by default, the Pod log of the Router is not able to see it.

Instead of setting up a rsyslog server outside of the cluster, this paper explores a “cloud-native” way of checking the HAproxy logs for debugging purposes.

Identify the container

The image of rsyslog/syslog_appliance_alpine seems like a good fit.

Customize the config file, rsyslog.conf, as below

global(processInternalMessages="on")
module(load="imrelp")
module(load="imptcp")
module(load="imudp" TimeRequery="500")
module(load="omstdout")
module(load="omelasticsearch")
module(load="mmjsonparse")
module(load="mmutf8fix")
input(type="imptcp" port="514")
input(type="imudp" port="514")
input(type="imrelp" port="1601")
syslog.* :omstdout:
include(file="/config/droprules.conf" mode="optional")  # this permits the user to easily drop unwanted messages
action(name="main_utf8fix" type="mmutf8fix" replacementChar="?")
# dump the haproxy log to console
local1.* :omstdout:

For debugging purposes, we emit the haproxy log to the stdout only.

Create a ConfigMap in the OpenShift cluster to host the config file.

oc new-project rsyslog
oc create cm cm-rsyslog-config --from-file=rsyslog.conf=rsyslog.conf

Create the deployment

Create the following deployment, apply it in the rsyslog namespace

apiVersion: apps/v1
kind: Deployment
metadata:
  name: rsyslog
  labels:
    app: rsyslog
spec:
  selector:
    matchLabels:
      app: rsyslog
  template:
    metadata:
      labels:
        app: rsyslog
    spec:
      securityContext:
        runAsUser: 0
      containers:
      - name: rsyslog
        image: rsyslog/syslog_appliance_alpine
        ports:
        - containerPort: 514
        env:
        - name: RSYSLOG_CONF
          value: /myconfig/rsyslog.conf
        volumeMounts:
        - mountPath: /myconfig
          name: rsyslog-config
        - mountPath: /config
          name: config
        - mountPath: /work
          name: work
        - mountPath: /logs
          name: logs
    volumes:
      - name: rsyslog-config
        configMap:
          name: cm-rsyslog-config
      - name: config
        emptyDir: {}
      - name: work
        emptyDir: {}
      - name: logs
        emptyDir: {}

Mount the volume for /config, /work, /logs with emptyDir. Mount the config file from the configMap. Use the environment variable to point to the config.

As it will listen on port 514, we need to assign the pod to run as root. Set the security context in the pod. In addition, assign the anyuid SCC (Security Context Constraints) to the default service account.

oc adm policy add-scc-to-user anyuid -z default

The deployment pod is running.

Expose service

Expose the rsyslog service within the cluster by using the default ClusterIP. Apply the following yaml file,

apiVersion: v1
kind: Service
metadata:
  name: rsyslog
spec:
  selector:
    app: rsyslog
  ports:
    - name: udp
      protocol: UDP
      port: 514
      targetPort: 514

Now the rsyslog will be available in the cluster at the address of rsyslog.rsyslog.svc

View Router’s log

Update the router’s environment setting,

oc -n default set env dc/router ROUTER_SYSLOG_ADDRESS=rsyslog.rsyslog.svc ROUTER_LOG_LEVEL=debug

When the router pod is restarted, the HAproxy’s log is available by check the rsyslog pod stdout,

oc -n rsyslog log rsyslog-675c46ff6f-hnrww -f

A sample is shown below,