Backup and Restore a Kafka Cluster on Kubernetes

The lastest Mirror Maker2 is able to replicate the Kafka from one cluster to the destination cluster. However the backup and restore requirement is still there for the local Kafka cluster. When a Kafka cluster is running on Kubernetes, the traditional backup/restore method needs to be revised. On the other hand, the standardization of Kubernetes Storage API with Container Storage Interface (CSI) makes the backup/restore for stateful app on Kubernetes much easier.

Using IBM event streams V10.1 (Kafka 2.6) as an example, this paper explores how we can backup and restore for a local Kafka Cluster. The Statefulsets of Kafka is running on Ceph RBD block storage with the Rook operator.

Backup Kafka Cluster with CSI Volume Cloning

Since Kafka is running on Kubernetes, the backup of the configuration of the Kafka cluster becomes a question of how to backup the etcd database, which is well known and I skip it here.

Now for the messages and offsets of the topics in the Kafka cluster, we need to back up the persistent volume for each of the brokers. In my test environment, I have 3 brokers with the Ceph RBD block storage provisioned dynamically with the PVC named as data-minimal-prod-kafka-[012] respectively.