Why I cannot connect to Kafka from outside?

Question

I am running kafka on ec2 instance. So amazon ec2 instance has two ips one is internal ip and second one is for external use.

I created producer from local machine, but it redirect to internal ip and give me connection unsuccessful error. Can anybody help me to configure kafka on ec2 instance, so that I can run producer from local machine. I am tried many combinations but didn't work.

Answer 1

In the Kafka FAQ (updated for new properties) you can read:

When a broker starts up, it registers its ip/port in ZK. You need to make sure the registered ip is consistent with what's listed in bootstrap.servers in the producer config. By default, the registered ip is given by InetAddress.getLocalHost.getHostAddress(). Typically, this should return the real ip of the host. However, sometimes (e.g., in EC2), the returned ip is an internal one and can't be connected to from outside. The solution is to explicitly set the host ip and port to be registered in ZK by setting the advertised.listeners property in server.properties.

Answer 2

SSH to your EC2 instance or wheverver you're hosting Kafka.

sudo nano /etc/hosts

Add:

127.0.0.1 <your-host-name> localhost

In my case it's:

127.0.0.1 ec2-12-34-56-78.ap-southeast-1.compute.amazonaws.com

Save and exit.

Answer 3

If you want to access from LAN, change following 2 files-

1. In config/server.properties:

   advertised.listeners=PLAINTEXT://server.ip.in.lan:9092
   

2. In config/producer.properties:

   bootstrap.servers=server.ip.in.lan:9092
   

In my case, the server.ip.in.lan value was 192.168.15.150

Answer 4

Below are the steps to connect Kafka from outside of EC2 instance.

1. Open Kafka server properties file on EC2.

   /kafka_2.11-2.0.0/config/server.properties

2. Set the value of advertised.listeners to

   advertised.listeners=PLAINTEXT://ec2-xx-xxx-xxx-xx.compute-1.amazonaws.com:9092

   This should be your Public DNS (IPv4) of EC2 instance.

3. Stop Kafka server.

4. Start Kafka server to see above configuration changes in action.

5. Now you can connect to your Kafka of EC2 instance from outside or from your localhost.

   Tried and tested on kafka_2.11-2.0.0

Answer 5

The easiest way how to reach your Kafka server (version kafka_2.11-1.0.0) on EC2 from consumer in external network is to change the properties file

kafka_2.11-1.0.0/config/server.properties

And modify the following line

listeners=PLAINTEXT://ec2-XXX-XXX-XXX-XXX.eu-central-1.compute.amazonaws.com:9092

Using your public address

Verified on 2.11-2.0.0

Answer 6

For EC2 you should edit the /etc/hosts file to add:

XXX.XXX.XXX.XXX ip-YYY-YYY-YYY-YYY

where XXX... is your external IP and the ip-YYY-YYY-YYY-YYY is the string returned by the hostname command. You can use 127.0.0.1 instead of your external IP to communicate inside the server.

host.name is deprecated - as are advertised.host.name and advertised.port

Answer 7

I solved this problem, by setting advertised.host.name in server.properties and metadata.broker.list in producer.properties to public IP address and host.name to 0.0.0.0.

Answer 8

I just did this in AWS. First get the Kafka server to listen on the correct interface/IP using host.name. For your case this would be the internal IP, not localhost, since your intent is for outside Kafka clients to connect. Any local clients will need to use that same address, not localhost.

Then set advertised.host.name to a host name, not an IP address. The trick is to get that host name to always resolve to the correct IP for both internal and external machines. I use /etc/hosts inside and DNS outside. See my full answer about Kafka and name resolution here.