Can't connect to kafka on my vm from a java client

Question

Goal: I am trying to send messages to kafka from a java cleint.
And it has been a pain..
Let me describe in brief.

1. I have installed kafka on a centos VM.
2. I ran the zookeeper that comes with it, the server , the producer and the client with all of the default properties files.
   I sent and received messages successfully.
3. I have a telnet connection from my computer to the zookeeper (2181) port and the kafka server (9092) port at the VM.

Now, I want write java code to send messages to the topic. I use the example from the quick start at the site:

Properties props = new Properties();
props.put("zk.connect", "http://XX.XX.XX.XX:2181"); //  where X is the ip
props.put("serializer.class", "kafka.serializer.StringEncoder");
producer = new Producer<String, String>(new ProducerConfig(props));

and it fails on the fourth line with the following excetptions :

kafka.common.FailedToSendMessageException: Failed to send messages after 3 tries

and

rg.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 400

Problems.

1. The exception.
   The bad parameters:
   In the kafka quick start example I see that it needs only zk.connect, serializer.class. when I run it it yells it needs metadata.broker.list in the constructor of the Producer.
   Does it? So I feel the ip and port of the kafka server.
   and btw - is it zk.connect or zookeeper connect?ZkTimeoutException: Unable to connect to zookeeper server within timeout: 400

2. maven bad versions
   I go to the site, i see that the latest version is kafka_2.8.0-0.8.0.
   problem no 1 - I download it using intelij (I think it is maven central) -
   I get all related jars - only that the kafka jars are empty (contain only manifest).

   problem no 2 - there are later versions then the one in the site. Are they official.
   any way, I downloaded org.apache.kafka:kafka_2.10:0.8.0

3. Using wireshark I see three SYN & ACK (triple handshake) and than and then FYN & ACK
   right after. in the logs of the zookeeper I see the following [2014-02-27 01:43:42,127] WARN EndOfStreamException: Unable to read additional data from client sessionid 0x0, likely client has closed socket
   (org.apache.zookeeper.server.NIOServerCnxn)
   Which means that I close the connection. Why?

Answer 1

In my situation I was trying to connect from to a broker hosted on an HDP 2.2 Sandbox VM and Azure CentOS 6.6 VMs in a cloud service and could not connect.

I was running a simple Java example (similar to the code snippets posted in the question) from my Mac.

What finally worked for me was to add the following parameters to the ./kafka/config/server.properties:

advertised.host.name advertised.port

For Azure I had to specify the cloud service name, so for example:

advertised.host.name=my-cloud-service.cloudapp.net advertised.port=6667

Additionally for Azure I had to make sure that port 6667 was opened as and endpoint.

Once I configured these I was able to run my Java code from my workstation and see the output from the consumer process running on the VM.

What lead me to this conclusion was the following URL:

https://cwiki.apache.org/confluence/display/KAFKA/FAQ

Look at: Why can't my consumers/producers connect to the brokers?

Answer 2

Kafka 0.8 does not need zk.connect params.
You have probably followed the 0.7 quickstart dodumentation. Check which version you are using , even if are you working with the old 0.7 distribution you DONT need "http" inside your zk.connect properties. Remove that as "xx.xxx.xx.xx:9092" ...
should be just

    props.put(“zk.connect”, “127.0.0.1:2181”);

for 0.8 follow this link, there are few changes the way producer works in these two different version. If you are new to Kafka then you should be using the latest version as there are lot of fixes in that