CODEWITHSUNDEEP
apache-kafkacloudera-manager
Rob Koch
Rob Koch

Reputation: 1583

Cloudera Kafka command line with boostrap-server not working

Looking through the instructions -- https://www.cloudera.com/documentation/kafka/latest/topics/kafka_command_line.html

I'm running these test command lines and one set works, but the other set doesn't.

Following instructions, it works, but noticed it has "zookeeper" as a parameter and I thought it was discontinued.

Producer:

/usr/bin/kafka-console-producer --broker-list local-ip:9092 --topic test

Consumer:

/usr/bin/kafka-console-consumer --bootstrap-server local-ip:9092 --topic test --from-beginning

the above doesn't work on the Cloudera version, but works on my standalone Kafka installs.

This works on Cloudera:

/usr/bin/kafka-console-consumer --zookeeper local-ip:2181 --topic test --from-beginning

Trying to understand what the difference between the Cloudera's Kakfa version (3.0.0-1.3.0.0.p0.40?) and mine (2.11-0.11.0.1) or there has to be something turned on or off.

I see some similar topic, and tried following them to no avail. I think it's something to do with Cloudera.

Upvotes: 2

Views: 995

Answers (1)

Daniel Argüelles
Daniel Argüelles

Reputation: 2349

Updated answer:

In my case, I have two brokers configured and the kafka's config value for offsets.topic.replication.factor set to 3. So, when Kafka tries to build a topic with more replicas than available brokers, an exception is thrown and the topic is not created.

The solution is to set offsets.topic.replication.factor = 2 and try again. Maybe you need to remove and deploy the brokers again.

I don't know why, maybe is a bug in Cloudera's Kafka release, but I solved it with a local kafka test.

I've downloaded latest version of Kafka from https://kafka.apache.org/downloads and updated the broker config file config\server.properties to use remote zookeeper server. With this, I had a mix configuration broker cluster:

  • brokers in my laptop
  • zookeeper in the cloudera cluster

With this configuration, I created a topic and run the kafka-console-consumer and kafka-console-producer from my laptop but against remote zookeeper:

$ kafka-topics --create --zookeeper zookeeper.cloudera-cluster:2181 --replication-factor 1 --partitions 1 --topic test
$ kafka-console-consumer --broker-list localhost:9092 --topic test
$ kafka-console-producer --broker-list localhost:9092 --topic test

this works propertly. Furthermore, using this the topic __consumer_offsets has been created automatically and now the new-consumer version works perfectly. At this point, you can remove the topic created and stop the local brokers and start to use kafka cluster normally.

Is this a bug from Cloudera's release? Maybe the version of Cloudera is not able to build __consumer_offsets automatically?

  • Kafka version downloaded: kafka_2.11-1.0.0.tgz
  • Cloudera's kafka version: 3.0.0-1.3.0.0.p0.40

Upvotes: 2

Related Questions