Volume snapshot is one way to backup, and restore Kafka data, but it's at the storage layer where the process itself doesn’t have any understanding of the application.
With Kafka MirrorMaker2 we can perform the backup and restore in a more flexible way. Let’s check this out.
I have a Kafka cluster with 3 brokers running with IBM eventstreams on top of OpenShift. Meantime, I have a single nodes K3s cluster where the Kafka is set up using the Strimizi Operator.
Using MirrorMaker2, we will first backup the topic data, and the offset for its consumer groups into the K3s Kafka cluster.
Simulating the disaster recovery, we will delete the topic and then restore the data from the K3s back to the OCP cluster using mirrormaker2. Lastly, we resume the MirrorMake2 data flow from OCP cluster to K3s.
The Kafka on the OCP is setup with the IBM eventstream operator, the Kafka portion of the YAML is shown below,
- name: external
- name: tls
We have turned off the auto topic creation. The external listener for Kafka is SASL_SSL based.
The Kafka cluster on the K3s is created with the native Strimizi operator. The YAML custom resource is listed below,