Perform Monitoring Task for Kafka with Golang

Image by Monica Volpin from Pixabay

Long time we have to rely on the Kafka command line tools to check the topic offset, to list the group lags and so on. You must have a JRE and install the Kafka in your system in the first place.

Recently, I discovered “franz-go” — A complete Apache Kafka client written in Go. With its pure go implementation and the full coverage of Kafka features we can create our own administrative tools to achieve some of the monitoring requirement.

Let’s use the adm package for the admin client.

go get github.com/twmb/franz-go/pkg/kadm

SASL Authentication

All my Kafka system is SASL authenticated. Let’s create a utility function to initiate a Kafka client with SASL.

Create a tls.Config with the CA cert from Kafka cluster.

func tlsConfigWithRootCA(pemFile string) (*tls.Config, error) {
 rootCA, err := x509.SystemCertPool()
 if err != nil {
  log.Printf("Failed to load system cert:%v", err)
  return nil, err
 }

 if rootCA == nil {
  log.Printf("root ca is nil. Creating new cert pool")
  rootCA = x509.NewCertPool()
 }

 cert, err := os.ReadFile(pemFile)
 if err != nil {
  log.Printf("Failed to read the cert: %v", err)
  return nil, err
 }
 ok := rootCA.AppendCertsFromPEM(cert)
 if !ok {
  log.Printf("The cert of %s…