CODEWITHSUNDEEP
apache-kafkakafka-consumer-apikafka-cluster
lakshmiravali rimmalapudi
lakshmiravali rimmalapudi

Reputation: 533

Not able to read messages from kafka consumer in kafka cluster setup

I have created two kafka brokers in a kafka cluster. When one broker is down I am not able to get any data to kafka consumer. I am using this command to read messages from consumer:

bin/kafka-console-consumer.sh --topic test_kafka_cluster \
    --bootstrap-server 127.0.0.1:9092,127.0.0.2:9092 --from-beginning

Upvotes: 0

Views: 861

Answers (2)

fvaleri
fvaleri

Reputation: 777

There are two parameters that affect topics availability for a consumer:

  • min.insync.replicas (minISR): Minimum number of in-sync partition's replicas.
  • replication.factor (RF): Total number of partition's replicas.

If you want your consumer to survive a broker outage, then you must have RF > minISR. With RF=2 and minISR=2 you can't tolerate any broker down, with RF=3 and minISR=2 you can tolerate 1 broker down, with RF=5 and minISR=2 you can tolerate 3 brokers down, and so on.

Note that the internal __consumer_offsets topic is used to store consumer offsets and has a default RF value of 3, which is not achievable in your cluster of 2 nodes. So you also need to set offsets.topic.replication.factor=1 at the cluster level.

Upvotes: 0

shasr
shasr

Reputation: 375

Here as per your console consumer configuration, IP address used here are 127.0.0.1 and 127.0.0.2 and two bootstrap servers are configured as 9092.

Verify both the ip's are reachable

bin/kafka-console-consumer.sh --topic test_kafka_cluster \
--bootstrap-server 127.0.0.1:9092,127.0.0.2:9092 --from-beginning

Ideally when we run tow kafka broker instance it will be running in two different ports.

Presuming Kafka is running in local Eg: localhost:9092 localhost:9093

Kafka instances running on two different host: Eg: 127.0.0.3:9092, 127.0.0.2:9092

If Kafka is running on docker/docker toolbox:

Console consumer on Docker toolbox:

docker exec <container-name> kafka-console-consumer --bootstrap-server 192.168.99.100:9093 --topic <topic-name> --from-beginning

Console consumer on Docker:

docker exec <container-name> kafka-console-consumer --bootstrap-server localhost:9093 localhost 9092 --topic <topic-name> --from-beginning

Upvotes: 1

Related Questions