Unable to run console consumer using a Kafka broker inside a Docker container

Question

I'm having a problem running a Kafka broker in a Docker container.

I have downloaded and unpacked a tar archive of Kafka 2.12-2.4.1.

I can run Zookeeper and Kafka brokers from the command line, producing and consuming messages successfully. When I build a docker container with Zookeeper, I can run that with a Kafka broker running from the command line, again producing and consuming messages successfully. When I put the broker into a docker container, the consumers and producers can't find a broker to connect to.

I have checked to ensure that Zookeeper has a connected broker: the zookeeper "dump" command below

echo dump | nc localhost 2181

returns

SessionTracker dump:
Session Sets (3)/(1):
0 expire at Thu May 28 09:53:33 GMT 2020:
0 expire at Thu May 28 09:53:36 GMT 2020:
1 expire at Thu May 28 09:53:39 GMT 2020:
    0x10002dfed700008
ephemeral nodes dump:
Sessions with Ephemerals (1):
0x10002dfed700008:
    /controller
    /brokers/ids/0
Connections dump:
Connections Sets (2)/(2):
0 expire at Thu May 28 09:53:37 GMT 2020:
2 expire at Thu May 28 09:53:47 GMT 2020:
    ip: /172.18.0.1:55656 sessionId: 0x0
    ip: /172.18.0.5:48474 sessionId: 0x10002dfed700008

which, to my untrained eye, looks like Zookeeper has a registered broker.

I can successfully list topics with the command

kafka-topics.sh --list --zookeeper localhost:2181

and when I run the command

netstat -plnt

I can see that I have listeners on ports 2181 and 9092, as below

tcp6       0      0 :::9092                 :::*                    LISTEN      54661/docker-proxy
tcp6       0      0 :::2181                 :::*                    LISTEN      47872/docker-proxy 

However, when I try to connect a consumer with the command

kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic aaaa

I get errors

WARN [Consumer clientId=consumer-console-consumer-8228-1, groupId=console-consumer-8228] Bootstrap broker localhost:9092 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)

which would seem to indicate that the client can't contact the broker.

I've tried several different variations on the listeners and advertised.listeners in the broker config file, setting them to localhost and 127.0.0.1, but to no avail.

I'm sure that I'm missing something simple: can anyone help?

Answer 1

So, I have solved the problem and I now have a working Docker/Kafka environment: I have a zookeeper, a broker and a consumer running in docker, and I can produce messages both from a producer running in docker and a consumer running from a command line.

As @GiorgosMyrianthous suspected, the problem was with the listeners and advertised listeners. I entered the following values in the Kafka broker properties file

listeners=PLAINTEXT://:29092,PLAINTEXT_HOST://:9092 advertised.listeners=PLAINTEXT://server:29092,PLAINTEXT_HOST://localhost:9092 listener.security.protocol.map=PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT

and I started the container with port 9092 exposed to the outside world.

I ran the Docker consumer with the bootstrap server set to server:29092.

I ran the Docker producer with the bootstrap server set to server:29092.

I ran the command line producer with the bootstrap server set to localhost:9092.

Messages typed in either of the producers appeared in the consumer.

Answer 2

Make sure to configure the following:

KAFKA_LISTENERS: LISTENER_INTERNAL://kafka-host:29092,LISTENER_EXTERNAL://kafka-host:9092
KAFKA_ADVERTISED_LISTENERS: LISTENER_INTERNAL://kafka-host:29092,LISTENER_EXTERNAL://localhost:9092
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: LISTENER_INTERNAL:PLAINTEXT,LISTENER_EXTERNAL:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: LISTENER_INTERNAL

---

• Now all your clients that run within the Docker Network, should use

  LISTENER_INTERNAL://kafka-host:29092
  

• All clients outside the Docker Network should use

  LISTENER_EXTERNAL://kafka-host:9092
  

---

For a more comprehensive guide, read @RobinMoffatt article.