CODEWITHSUNDEEP
apache-kafkakafka-consumer-apikafka-producer-api
Ajay Misra
Ajay Misra

Reputation: 210

Kafka mirror maker not working

I am facing a weird problem where my mirror-maker is able to mirror the topic name in the destination cluster but I am seeing no messages when I try to consume messages from it. Below is my configuration file -

producer.config

bootstrap.servers=dest_broker_ip:9092,dest_broker_ip:9092
acks=1
batch.size=100
max.in.flight.requests.per.connection=1
retries=1000000
client.id=mirror_maker_producer

consumer.config

bootstrap.servers=source_broker_ip:9092,source_broker_ip:9092,source_broker_ip:9092    
exclude.internal.topics=true
group.id=test-consumer-group_3
client.id=mirror_maker_consumer

I then start my mirror maker with the following command

bin/kafka-run-class.sh kafka.tools.MirrorMaker --consumer.config config/consumer.config --producer.config config/producer.config --whitelist=".*"

Once the mirror maker is started I go to my source cluster and create a topic along with some messages with below command

bin/kafka-verifiable-producer.sh --topic u39 --max-messages 2000 --broker-list 10.******:9092

this will now create a topic u39 and will push 2000 messages to it. Now if I run consumer on the same source cluster I am able to consume the messages which confirms messages were produced at source cluster

I now go to my destination cluster and try to create a topic with the same name i.e. u39 this returns an error stating topic already exists which confirms that mirror maker replicated the topic from source cluster to destination cluster but when I try to consume messages on the destination cluster by below command

bin/kafka-console-consumer.sh --zookeeper 10.200.14.117:2181 --topic u39 --from-beginning

I don't see any messages. Not sure what is wrong any help/direction is greatly appreciated.

Update 1 - I am using Kafka 1.0.0

Update 2 - Both the clusters are completely isolated the setup is on AWS one cluster is in east-1 and the other is in west-2. Again individually I am able to produce and consume messages on both the clusters.

Upvotes: 1

Views: 3211

Answers (1)

Terra Field
Terra Field

Reputation: 71

I think you're mixing syntax from different versions of the console consumer. The default should now be the new consumer and the new consumer will want a bootstrap server instead of a ZooKeeper.

Try:

kafka-console-consumer.sh --bootstrap-server 10.******:9092 --topic u39 --from-beginning

Upvotes: 1

Related Questions