Connecting Kafka running on EC2 machine from my local machine

Question

I am new to Kafka and searched different posts in the forums but couldn't find the solution. I have installed kafka on an EC2 instance and trying to connect the same from my ubuntu local machine. My objective is to have python kafka clients(both Producer and Consumer) running on my local machine and sending/receiving data via EC2 kafka instance. Is that possible?

Properties set in server.properties config file:

listeners=PLAINTEXT://0.0.0.0:9092 
advertised.listeners=PLAINTEXT://<ec2-public-DNS>:9092

On Kafka EC2 Instance:

netstat -an | grep LISTEN 
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 
tcp6 0 0 :::9092 :::* LISTEN 

On Zookeeper cli on Kafka EC2 Instance:

get /brokers/ids/0
{"listener_security_protocol_map":{"PLAINTEXT":"PLAINTEXT"},"endpoints":["PLAINTEXT://<ec2-public-DNS>:9092"],"jmx_port":-1,"host":"<ec2-public-DNS>","timestamp":"1492900361516","port":9092,"version":4}
cZxid = 0xed
ctime = Sat Apr 22 22:32:41 UTC 2017
mZxid = 0xed
mtime = Sat Apr 22 22:32:41 UTC 2017
pZxid = 0xed
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x15b97cb9d060000
dataLength = 250
numChildren = 0

Python client(Producer) on my local machine:

from kafka import KafkaProducer
import time
import json

producer = KafkaProducer(bootstrap_servers="<ec2-public-DNS>:9092")
for i in range(100):
    dict = {}
    dict['name_'+str(i)] = 'FILE_' + str(i)
    dict['size_'+str(i)] = '23.' + str(i)
    dict['host_'+str(i)] = '10.0.0.0' + str(i)
    jd = json.dumps(dict)
    producer.send('console-test-topic', jd)
    time.sleep(2)

Python client(Consumer) on my local machine:

from kafka import KafkaConsumer

consumer = KafkaConsumer('console-test-topic', bootstrap_servers="<ec2-public-DNS>:9092")
for msg in consumer:
    print (msg)

But, the producer is not able to connect to Kafka EC2 instance and failing with below error:

**kafka.errors.NoBrokersAvailable: NoBrokersAvailable** 

Please refer to the link for my security group rules:

goo.gl/ZUVknv

Running Producer in debug mode on my local machine:

DEBUG:kafka.producer.kafka:Starting the Kafka producer
DEBUG:kafka.metrics.metrics:Added sensor with name connections-closed
DEBUG:kafka.metrics.metrics:Added sensor with name connections-created
DEBUG:kafka.metrics.metrics:Added sensor with name select-time
DEBUG:kafka.metrics.metrics:Added sensor with name io-time
INFO:kafka.client:Bootstrapping cluster metadata from [('ec2-54-91-87-14.compute-1.amazonaws.com', 9092, 0)]
DEBUG:kafka.client:Attempting to bootstrap via node at ec2-54-91-87-14.compute-1.amazonaws.com:9092
DEBUG:kafka.metrics.metrics:Added sensor with name bytes-sent-received
DEBUG:kafka.metrics.metrics:Added sensor with name bytes-sent
DEBUG:kafka.metrics.metrics:Added sensor with name bytes-received
DEBUG:kafka.metrics.metrics:Added sensor with name request-latency
DEBUG:kafka.metrics.metrics:Added sensor with name node-bootstrap.bytes-sent
DEBUG:kafka.metrics.metrics:Added sensor with name node-bootstrap.bytes-received
DEBUG:kafka.metrics.metrics:Added sensor with name node-bootstrap.latency
DEBUG:kafka.client:Node bootstrap connected
DEBUG:kafka.cluster:Updated cluster metadata to ClusterMetadata(brokers: 1, topics: 2, groups: 0)
INFO:kafka.client:Bootstrap succeeded: found 1 brokers and 2 topics.
DEBUG:kafka.client:Initiating connection to node 0 at ec2-54-91-87-14.compute-1.amazonaws.com:9092
DEBUG:kafka.metrics.metrics:Added sensor with name node-0.bytes-sent
DEBUG:kafka.metrics.metrics:Added sensor with name node-0.bytes-received
DEBUG:kafka.metrics.metrics:Added sensor with name node-0.latency
INFO:kafka.producer.kafka:Kafka producer closed
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/dist-packages/kafka/producer/kafka.py", line 335, in __init__
    **self.config)
  File "/usr/local/lib/python2.7/dist-packages/kafka/client_async.py", line 210, in __init__
    self.config['api_version'] = self.check_version(timeout=check_timeout)
  File "/usr/local/lib/python2.7/dist-packages/kafka/client_async.py", line 828, in check_version
    raise Errors.NoBrokersAvailable()
kafka.errors.NoBrokersAvailable: NoBrokersAvailable

I tried with running producer client in another EC2 instance(in the same VPN as kafka instance) and it is working fine. But, it is not working when producer is running on my local machine. Does 'advertised.listeners' property advertise the kafka brokers just in the same(AWS VPN) network? Or can I also connect it from my local machine? Please let me know if someone can point me in the right direction.

Answer 1

I went through a similar instead some months ago, basically I have an ec2 instance with Kafka in N. Virginia and I configured topbeat on my local machine to send metrics to that ec2 instance. I was able to get it working by adding

advertised.host.name=public-ip

as configuration in the server.properties of kafka, but according to the documentation this properties is deprecated.

Reading further documentation, it is said that if you are in an IaaS environment, you have to configure the advertised.listeners different from the interface to which the broker binds.

Answer 2

Did you by any chance neglect to open inbound port 9092 in your AWS security group to your local network's external IP? Open it to 0.0.0.0/0 if you elect to open to all (be cautioned about security implication though).