CODEWITHSUNDEEP
dockerapache-kafkaapache-zookeeper
Dogukan Levendoglu
Dogukan Levendoglu

Reputation: 81

How to scale kafka and zookeeper with docker-compose?

I'm all new to kafka and zookeeper. I'm trying to run 3 zookeeper nodes and 3 kafka brokers with docker-compose's --scale functionality. But somehow zookeeper instances won't sync (I guess) and brokers will end up in different clusters. Sometimes 2-1, sometimes 1-1-1. Either way they will not be aware of each other because when I try to create a topic with 3 partitions and 3 replications it will give me an error saying the number of available brokers are less than the replication factor. This is not the case when I don't scale zookeeper but only kafka with docker-compose up --scale kafka=3. Then I can see with kafka-cluster.sh that all of the brokers are in the same cluster.

Here is my docker-compose.yml

version: '3'
services:
  zookeeper:
    image: bitnami/zookeeper:latest
    environment:
      ALLOW_ANONYMOUS_LOGIN: "yes"
    
  kafka:
    image: bitnami/kafka:latest
    depends_on:
      - zookeeper
    environment:
      KAFKA_CFG_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_CFG_ADVERTISED_LISTENERS: PLAINTEXT://:9092
      ALLOW_PLAINTEXT_LISTENER: "yes"

At the end of the day what I'm trying to achive is run 3 zookeepers, 3 brokers, and one nifi instance to publish and consume messages with from kafka. I know having 3 instances of kafka and zookeeper on the same machine doesn't make sense, but I'm trying to understand how this all works.

Thank you!

Upvotes: 0

Views: 2742

Answers (1)

OneCricketeer
OneCricketeer

Reputation: 191738

It's not possible to use the scale CLI because all brokers will end up with with same broker.id value, and conflict with each other in Zookeeper.

You'll need to define three separate brokers (you don't need 1 Zookeeper per broker) in the YAML

e.g.

Or instead use something like Minikube with the Strimzi Kafka operator that can appropriately define container replicas. There's also probably a Nifi Helm Chart or Operator that exists.

Upvotes: 1

Related Questions