CODEWITHSUNDEEP
apache-kafka
user2381482
user2381482

Reputation: 29

Kafka consumer and producer throwing error

I am getting below error while trying to pass words in Kafka-consumer, commands with i entered

console-1:(for producer)

    export PATH=$PATH:/usr/hdp/current/kafka-broker/bin

kafka-topics.sh --create --zookeeper ip-172-31-20-58.ec2.internal:2181 --replication-factor 1 --partitions 1 --topic testuday1234

kafka-console-producer.sh --broker-list ip-172-31-20-58.ec2.internal:6667 --topic testuday1234

console-2: (for consumer)

export PATH=$PATH:/usr/hdp/current/kafka-broker/bin
kafka-console-consumer.sh --zookeeper localhost:2181 --topic testuday1234 --from-beginning

Please help me in resolving these error

Error i am getting in Producer console:

[udaychitukula6587@ip-172-31-38-183 ~]$ kafka-console-producer.sh --broker-list ip-172-31-20-58.ec2.internal:6667 --topic testuday1234
hi
[2018-05-28 15:27:36,761] WARN Error while fetching metadata [{TopicMetadata for topic testuday1234 ->
No partition metadata for topic testuday1234 due to kafka.common.LeaderNotAvailableException}] for topic [testuday1234]: class kafka.common.LeaderNotAvailableExcep
tion (kafka.producer.BrokerPartitionInfo)
[2018-05-28 15:27:36,774] WARN Error while fetching metadata [{TopicMetadata for topic testuday1234 ->
No partition metadata for topic testuday1234 due to kafka.common.LeaderNotAvailableException}] for topic [testuday1234]: class kafka.common.LeaderNotAvailableExcep
tion (kafka.producer.BrokerPartitionInfo)

Error i am getting in consumer console:

[udaychitukula6587@ip-172-31-38-183 ~]$ kafka-console-consumer.sh --zookeeper localhost:2181 --topic testuday123 --from-beginning
{metadata.broker.list=ip-172-31-20-58.ec2.internal:6667,ip-172-31-53-48.ec2.internal:6667,ip-172-31-60-179.ec2.internal:6667, request.timeout.ms=30000, client.id=c
onsole-consumer-63526, security.protocol=PLAINTEXT}
{metadata.broker.list=ip-172-31-20-58.ec2.internal:6667,ip-172-31-53-48.ec2.internal:6667,ip-172-31-60-179.ec2.internal:6667, request.timeout.ms=30000, client.id=c
onsole-consumer-63526, security.protocol=PLAINTEXT}

Upvotes: 0

Views: 1540

Answers (1)

Lalit
Lalit

Reputation: 2014

There is a couple of things that I've noted here.

First, in the newer versions (I think from 0.10.1) of Kafka for the console consumer, we need to use --bootstrap-server option as opposed to --zookeeper. Could you please confirm the version you're using? and also try to run the consumer command with --bootstrap-server option?

Second, for the producer in such a scenario, I would recommend checking 3 things to confirm where the issue might be:

  1. The leader of a Kafka cluster is elected by the zookeeper, so it might be worth checking by running a zookeeper-client shell to see if there is an active controller in the Kafka cluster (in the znode path - /brokers/ids/[brokerId]).
  2. Try running a Kafka-topics --describe --topic command to see if the topic has an active leader partition i.e. the Leader column in the output of the command should NOT have None. I've run into this myself before.
  3. The last one is about the port number of the broker, could you please check and confirm if the broker is actually listening on that port. You'll find this information (listeners and advertised.listeners) in server.properties file on the broker. I found this post that you might find useful where the user had a problem with the port 6667.

I hope this helps!

Upvotes: 1

Related Questions