CODEWITHSUNDEEP
apache-kafkakafka-consumer-api
Boris Burkov
Boris Burkov

Reputation: 14436

Kafka in docker-compose: topics not in preferred replica for broker

I am running a testing kafka "cluster" consisting of a single Kafka broker and a single Zookeeper node from docker-compose, using the following (official) docker-compose.yml:

---
    version: '2'
    
    services:
      zookeeper:
        image: confluentinc/cp-zookeeper:6.0.0
        hostname: zookeeper
        container_name: zookeeper
        ports:
          - "2181:2181"
        environment:
          ZOOKEEPER_CLIENT_PORT: 2181
          ZOOKEEPER_TICK_TIME: 2000
    
      broker:
        image: confluentinc/cp-kafka:6.0.0
        hostname: broker
        container_name: broker
        depends_on:
          - zookeeper
        ports:
          - "29092:29092"
        environment:
          KAFKA_BROKER_ID: 101
          KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
          KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
          KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker:9092,PLAINTEXT_HOST://localhost:29092
          KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
          KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
          KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
          KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0

After starting this "cluster" with docker-compose up I successfully create a topic using console command:

docker exec broker kafka-topics --create --topic myTopic --bootstrap-server localhost:29092 --replication-factor 1 --partitions 1

and programmatically create messages with a Golang producer, while listening to them with a console client:

docker exec broker kafka-console-consumer --from-beginning --bootstrap-server localhost:29092 --topic myTopic --timeout-ms 15000

Every once in a while both the console consumer and the Golang programmatic producer hang and stop connecting to kafka. When I look into the logs of Kafka broker to figure out, what's wrong, I see a wall of repeating messages of the following structure about topics being not in preferred replica:

broker       | [2021-05-09 06:08:50,144] INFO [Controller id=101] Processing automatic preferred replica leader election (kafka.controller.KafkaController)
broker       | [2021-05-09 06:08:50,149] TRACE [Controller id=101] Checking need to trigger auto leader balancing (kafka.controller.KafkaController)
broker       | [2021-05-09 06:08:50,165] DEBUG [Controller id=101] Topics not in preferred replica for broker 101 HashMap() (kafka.controller.KafkaController)
broker       | [2021-05-09 06:08:50,166] TRACE [Controller id=101] Leader imbalance ratio for broker 101 is 0.0 (kafka.controller.KafkaController)

Could you explain, what's going wrong? How could a wrong replica be assigned, if there's just 1 partition and 1 broker?

Upvotes: 4

Views: 12193

Answers (1)

OneCricketeer
OneCricketeer

Reputation: 191681

I don't think this small section of logs is your issue

Those logs say there's no topics not in preferred replica set... It's printing an empty Hashmap

Similarly, the imbalance ratio is zero, so the cluster is perfectly balanced

Upvotes: 2

Related Questions