Strimzi

Kafka

Cluster Rebalance

Operators

Ocp

Scaling Your Kafka Cluster

Rebalancing of partitions across brokers

Image by Herbert Aust from Pixabay

In “Day 2” operations, an increase in traffic or data volume often necessitates scaling up your Kafka cluster. However, simply adding new brokers to the environment isn’t enough; the existing data and partitions do not automatically migrate. To ensure your workload is effectively distributed across the new hardware, you must perform specific administrative tasks. This paper examines these steps in detail.

Testing Kafka Cluster

We set up a Kafka cluster on OpenShift with IBM Event Operator.

Create the Kafka Node Pools for controllers and brokers.

apiVersion: ibmevents.ibm.com/v1beta2
kind: KafkaNodePool
metadata:
  name: controller
  namespace: kafka-rebalance
  labels:
    ibmevents.ibm.com/cluster: my-cluster
  annotations:
    ibmevents.ibm.com/next-node-ids: "[0-99]"
spec:
  replicas: 3
  roles:
    - controller
  storage:
    type: jbod
    volumes:
      - id: 0
        type: persistent-claim
        size: 10Gi
        deleteClaim: false
---
apiVersion: ibmevents.ibm.com/v1beta2
kind: KafkaNodePool
metadata:
  name: broker
  namespace: kafka-rebalance
  labels:
    ibmevents.ibm.com/cluster: my-cluster
  annotations:
    ibmevents.ibm.com/next-node-ids: "[100-199]"
spec:
  replicas: 3
  roles:
    - broker…