CODEWITHSUNDEEP
dockerapache-kafka
Codious-JR
Codious-JR

Reputation: 1748

Accessing IP address of a docker container in a custom network

I am using docker compose to start kafka and zookeeper

my docker-compose is as follows:

version: '3'
services:
  zookeeper:
    image: zookeeper:3.4
    networks:
      - pipeline

  kafka:
    image: ches/kafka
    networks:
      - pipeline
    environment:
      - ZOOKEEPER_IP=zookeeper
    ports:
      - "9092:9092"
    links:
      - zookeeper:zookeeper
    depends_on:
      - zookeeper

networks:
  pipeline:

So this starts a container running zookeeper and starts a container running kafka server which successfully connects to zookeeper.

Additionally I can successfully create a kafka producer and a kafka consumer to communicate between each other by the following commands:

Producer:

docker run --rm -it --network pipeline_pipeline ches/kafka \
    kafka-console-producer.sh --topic pipeline-dev \
    --broker-list kafka:9092

Consumer:

docker run --rm --network pipeline_pipeline ches/kafka \
    kafka-console-consumer.sh --topic pipeline-dev \
    --from-beginning \
    --bootstrap-server kafka:9092

Now the problem I am facing is that in the config the ADVERTISED HOST on the kafka server is set to kafka, which prevents me from connecting to the kafka server from local.

I am using the kafka-python library to connect to kafka in the following way:

from kafka import KafkaConsumer

consumer = KafkaConsumer('pipeline-dev', bootstrap_servers='localhost:9092')

print 'starting to listen'
for msg in consumer:
    print msg

But here I get the error that Brokers are not available! That is because I have set the bootstrap servers as localhost:9092. Even though I have forwarded the ports, I am unable to connect to the kafka server because the advertised host name is kafka and not localhost.

Setting bootstrap servers as kafka:9092 does not work because the code is running in local and not in the docker network.

I am not sure how to solve this problem.

Upvotes: 0

Views: 1809

Answers (1)

Tarun Lalwani
Tarun Lalwani

Reputation: 146630

Make a host entry in your /etc/hosts like below

127.0.0.1 kafka

Above will work when you want to use your code on the docker host machine. If you want to use it on some other machine then you should use

<DockerHostIP> kafka

Basically you want to have the name resolved when the advertised name returns kafka

Upvotes: 2

Related Questions