CODEWITHSUNDEEP
hadoopapache-kafkahortonworks-data-platform
Samhash
Samhash

Reputation: 140

Kafka console producer Error in Hortonworks HDP 2.3 Sandbox

I have searched it all over and couldn't find the error. I have checked This Stackoverflow Issue but it is not the problem with me

  1. I have started a zookeeper server

Command to start server was

 bin/zookeeper-server-start.sh config/zookeeper.properties

  1. Then I SSH into VM by using Putty and started kafka server using

    $ bin/kafka-server-start.sh config/server.properties

  2. Then I created Kafka Topic and when I list the topic, it appears. enter image description here

  3. Then I opened another putty and started kafka-console-producer.sh and typed any message (even enter) and get this long repetitive exception. enter image description here enter image description here

Configuration files for zookeeper.properties, server.properties, kafka-producer.properties are as following (respectively) enter image description here

enter image description here

enter image description here

The version of Kafka i am running is 8.2.2. something as I saw it in kafka/libs folder. P.S. I get no messages in consumer.

Can any body figure out the problem? The tutorial I was following was [This][9]

8http://%60http://www.bogotobogo.com/Hadoop/BigData_hadoop_Zookeeper_Kafka_single_node_single_broker_cluster.php%60

Upvotes: 3

Views: 3619

Answers (3)

Chris Gerken
Chris Gerken

Reputation: 16390

Within Kafka internally there is a conversation that goes on between both producers and consumers (clients) and the broker (server). During those conversations clients often ask the server for the address of a server broker that's managing a particular partition. The answer is always a fully-qualified host name. Without going into specifics if you ever refer to a broker with an address that is not that broker's fully-qualified host name there are situations when the Kafka implementation runs into trouble.

Another mistake that's easy to make, especially with the Sandbox, is referring to a broker by an address that's not defined to the DNS. That's why every node on the cluster has to be able to address every other node in the cluster by fully-qualified host name. It's also why, when accessing the sandbox from another virtual image running on the same machine you have to add sandbox.hortonworks.com to the image's hosts file.

Upvotes: 0

asmaier
asmaier

Reputation: 11776

On the hortonworks sandbox have a look at the server configuration:

$ less /etc/kafka/conf/server.properties

In my case it said

...
listeners=PLAINTEXT://sandbox.hortonworks.com:6667
...

This means you have to use the following command to successfully connect with the console-producer

$ cd /usr/hdp/current/kafka-broker
$ bin/kafka-console-producer.sh --topic test --broker-list sandbox.hortonworks.com:6667

It won't work, if you use --broker-list 127.0.0.1:6667 or --broker-list localhost:6667 . See also http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.0/bk_installing_manually_book/content/configure_kafka.html

To consume the messages use 

$ bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-beginning

Upvotes: 5

Bector2
Bector2

Reputation: 152

As you mentioned in your question that you are using HDP 2.3 and for that when you are running Console-Producer
You need to provide sandbox.hortonworks.com:6667 in Broker-list.

Please use the same while running Console-Consumer.

Please let me know in case still you face any issue.

Upvotes: 0

Related Questions