Kafka - Simple consumer/producer setup doesn't work on different machines, but works locally

Question

I've installed and setup Kafka using the following simple producer/consumer tutorial:

https://kafka.apache.org/quickstart

I have two machines, and both are using Ubuntu.

Resume of the problem:

If I use the producer and consumer on the same machine, everything works fine. If I use the producer on machine 2, and the rest on machine 1, such as kafka, zookeeper server and the consumer, I never receive any messages on machine 1.

Machine 1 has IP: 192.168.1.100

Machine 2 has IP: 192.168.1.101

Working Example using just the Machine 1 only, with 4 console applications

Console 1 - started zookeeper:

bin/zookeeper-server-start.sh config/zookeeper.properties

Console 2 - started kafka server

bin/kafka-server-start.sh config/server.properties

Console 3 Created a topic called test:

bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test

Tested the topic:

bin/kafka-topics.sh --list --zookeeper localhost:2181

Started the consumer

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test

Console 4 - Send some messages:

bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test

It works!

NOT Working Example using Machine 1 and 2

Machine 1 - Console 1 - started zookeeper:

bin/zookeeper-server-start.sh config/zookeeper.properties

Machine 1 - Console 2 - started kafka server

bin/kafka-server-start.sh config/server.properties

Machine 1 - Console 3 Created a topic called test:

bin/kafka-topics.sh --create --zookeeper 192.168.1.100:2181 --replication-factor 1 --partitions 1 --topic test

Tested the topic:

bin/kafka-topics.sh --list --zookeeper 192.168.1.100:2181

Started the consumer

bin/kafka-console-consumer.sh --bootstrap-server 192.168.1.100:9092 --topic test

Machine 2 - Console 1 - Send some messages to kafka on the IP 192.168.1.100

bin/kafka-console-producer.sh --broker-list 192.168.1.100:9092 --topic test

Doesn't work ...

What am I missing here?

EDIT:

I've used now the kafkacat to test the connection...here is the output.

root@ubuntu:~/kafka/kafka_2.11-0.11.0.1# kafkacat -b 192.168.1.100 -t test -L
Metadata for test (from broker -1: 192.168.1.100:9092/bootstrap):
 1 brokers:
  broker 0 at ubuntu:9092
 1 topics:
  topic "test" with 1 partitions:
    partition 0, leader 0, replicas: 0, isrs: 0
%3|1507802180.807|FAIL|rdkafka#producer-1| [thrd:ubuntu:9092/0]: ubuntu:9092/0: Connect to ipv4#127.0.1.1:9092 failed: Connection refused
%3|1507802180.807|ERROR|rdkafka#producer-1| [thrd:ubuntu:9092/0]: ubuntu:9092/0: Connect to ipv4#127.0.1.1:9092 failed: Connection refused

Why is the 127.0.1.1:9092 set above? Shouldn't be 192.168.1.100:9092?

Answer 1

From the FAQ, a common problem is the hostname:

When a broker starts up, it registers its ip/port in ZK. You need to make sure the registered ip is consistent with what's listed in metadata.broker.list in the producer config. By default, the registered ip is given by InetAddress.getLocalHost.getHostAddress(). Typically, this should return the real ip of the host. However, sometimes (e.g., in EC2), the returned ip is an internal one and can't be connected to from outside. The solution is to explicitly set the host ip to be registered in ZK by setting the hostname property in server.properties. In another rare case where the binding host/port is different from the host/port for client connection, you can set advertised.host.name and advertised.port for client connection.

So, open your server.properties and change the hostname:

host.name=<your hostname>

If this still doesn't work, try to modify the advertised.host.name:

advertised.host.name=<your ip>

If this really does not work, have a look at the advertised.listeners and ensure it is either 0.0.0.0:port or <your.ip>:port. Example:

advertised.listeners=PLAINTEXT://0.0.0.0:9092