MirrorMaker2 for Data Recovery

Image by Sabine Zierer from Pixabay

Volume snapshot is one way to backup, and restore Kafka data, but it's at the storage layer where the process itself doesn’t have any understanding of the application.

With Kafka MirrorMaker2 we can perform the backup and restore in a more flexible way. Let’s check this out.

Test Environment

I have a Kafka cluster with 3 brokers running with IBM eventstreams on top of OpenShift. Meantime, I have a single nodes K3s cluster where the Kafka is set up using the Strimizi Operator.

Using MirrorMaker2, we will first backup the topic data, and the offset for its consumer groups into the K3s Kafka cluster.

Simulating the disaster recovery, we will delete the topic and then restore the data from the K3s back to the OCP cluster using mirrormaker2. Lastly, we resume the MirrorMake2 data flow from OCP cluster to K3s.

Kafka Clusters

The Kafka on the OCP is setup with the IBM eventstream operator, the Kafka portion of the YAML is shown below,

kafka:
  replicas: 3
  config:
    inter.broker.protocol.version: '3.2'
    log.cleaner.threads: 6
    log.message.format.version: '3.2'
    num.io.threads: 24
    num.network.threads: 9
    num.replica.fetchers: 3
    offsets.topic.replication.factor: 3…