CODEWITHSUNDEEP
pythondockerapache-kafkadocker-composedocker-toolbox
Patrick
Patrick

Reputation: 362

Writing to Kafka running in Docker from Python fails

I have Zookeeper and Kafka running in Docker on OS X using Docker Toolbox and Docker Compose. My docker-compose.yml file is:

zk:
  image: patrickmay/zookeeper:3.4.6-jessie
  ports:
   - "2181:2181"
  command: /usr/local/zookeeper/bin/zkServer.sh start-foreground
kafka0:
  image: patrickmay/kafka:0.8.2.1-jessie
  ports:
   - "9092:9092"
  command: /usr/local/bin/start-kafka.sh 0 docker-machine 9092
  links:
   - zk
kafka1:
  image: patrickmay/kafka:0.8.2.1-jessie
  ports:
   - "9093:9092"
  command: /usr/local/bin/start-kafka.sh 1 docker-machine 9093
  links:
   - zk
kafka2:
  image: patrickmay/kafka:0.8.2.1-jessie
  ports:
   - "9094:9092"
  command: /usr/local/bin/start-kafka.sh 2 docker-machine 9094
  links:
   - zk

I've created a topic called test_impressions:

~/packages/kafka_2.11-0.8.2.1/bin/kafka-topics.sh --zookeeper docker-machine --describe --topic test_impressions
Topic:test_impressions  PartitionCount:3    ReplicationFactor:2 Configs:
    Topic: test_impressions Partition: 0    Leader: 0   Replicas: 2,0   Isr: 0
    Topic: test_impressions Partition: 1    Leader: 1   Replicas: 0,1   Isr: 1
    Topic: test_impressions Partition: 2    Leader: 1   Replicas: 1,2   Isr: 1

I try to write to it with this simple Python script:

#!/usr//bin/env python

from __future__ import print_function
import sys
from kafka import SimpleProducer, KafkaClient

kafka = KafkaClient('docker-machine:9094')
producer = SimpleProducer(kafka)

producer.send_messages(b'test_impressions', b'some message')

I get this error:

Traceback (most recent call last):
  File "./kafka-producer.py", line 12, in <module>
    producer.send_messages(b'test_impressions', b'some message')
  File "/Users/patrick.may/projects/ad-server/frequency-cap/venv/lib/python2.7/site-packages/kafka/producer/simple.py", line 52, in send_messages
    partition = self._next_partition(topic)
  File "/Users/patrick.may/projects/ad-server/frequency-cap/venv/lib/python2.7/site-packages/kafka/producer/simple.py", line 36, in _next_partition
    self.client.load_metadata_for_topics(topic)
  File "/Users/patrick.may/projects/ad-server/frequency-cap/venv/lib/python2.7/site-packages/kafka/client.py", line 378, in load_metadata_for_topics
    kafka.common.check_error(topic_metadata)
  File "/Users/patrick.may/projects/ad-server/frequency-cap/venv/lib/python2.7/site-packages/kafka/common.py", line 233, in check_error
    raise error_class(response)
kafka.common.LeaderNotAvailableError:     TopicMetadata(topic='test_impressions', error=5, partitions=[])

I've tried with all three docker-machine ports (9092, 9093, and 9094). Do I have something misconfigured?

Thanks.

Upvotes: 0

Views: 775

Answers (1)

Patrick
Patrick

Reputation: 362

Found my error. Using docker-machine in the docker-compose.yml file was an error because docker-machine is not in that /etc/hosts file. Changing that to the numeric IP address fixed the problem.

Upvotes: 0

Related Questions