Running Kafka Exporter in OpenShift

Image by Karsten Bergmann from Pixabay

In my last blog, I run KEDA to manage the autoscaling based on the consumer group’s lag. Ideally, if we can see this through a Grafana Dashboard, that will be the best. However, a little bit of effort is required for this…

Turn on Kafka Exporter

Kafka exporter is a Prometheus exporter that can monitor Kafka topic, throughputs, consumer groups, and so on. It’s included in the Strimizi Operator and so does it for IBM event streams. To enable it for event streams, configure the CR with the following settings:

...
  strimziOverrides:
    kafkaExporter: {}
...

You will see an extra POD is running,

$ oc -n es get pods | grep exporter
es-kafka-exporter-69fc97684c-bkccn    1/1     Running   0          2d23h

If we check the POD logs, we can find many broker not connected error message as shown below.

time="2022-03-15T06:31:07Z" level=error msg="Cannot get offset of group eventstreams-apicurio-registry: kafka: broker not connected" source="kafka_exporter.go:384"
time="2022-03-15T06:31:07Z" level=error msg="Cannot get offset of group __strimzi-topic-operator-kstreams: kafka: broker not connected" source="kafka_exporter.go:384"
time="2022-03-15T06:32:07Z" level=error msg="Cannot get offset of group…