Why Kafka broker connects to itself?

Question

Few days ago I tried to configure Kafka Docker container with Docker Compose and port mapping and discover interesting behavior which I do not fully understand:

Kafka broker seems to connect to itself. Why ?

My set up is:

• Ubuntu 14.04, Docker 1.13.1, Docker-Compose 1.5.2
• Kafka 0.10 listens on port 9092, this port is exposed by container.
• In Docker Compose I have port mapping from container port 9092 to local port 4005.
• I configured host name of my Docker Host machine and local port from Compose in advertised.listeners (docker-host:4005) since broker should be visible from my company network.

With this set up when I try to send/fetch data to/from Kafka, all attempts end up with:

Topic metadata fetch included errors: {topic_name=LEADER_NOT_AVAILABLE}

After trying various combinations of ports and host names in advertised.listeners, I discovered that sole working combination is localhost:9092. Any attempt to change hostname or port led to the error mentioned above.
This made me think that Kafka tries to connect to address configured in advertised.listeners and this is somehow related to topic metadata.

So inside Docker container I did:

• redirect traffic to "docker-host" to loopback
  echo "127.0.0.1 $ADVERTISED_HOST" >> /etc/hosts
• configure Kafka to listen on all interfaces and port (exact as advertised)
  sed -r -i "s/#(listeners)=(.*)/\1=PLAINTEXT:\/\/0.0.0.0:4005/g" $KAFKA_HOME/config/server.properties
• advertise "docker-host" and external port
  sed -r -i "s/#(advertised.listeners)=(.*)/\1=PLAINTEXT:\/\/$ADVERTISED_HOST:4005/g" $KAFKA_HOME/config/server.properties

And now it works like a charm.

However I still do not understand:

1. Why Kafka broker might need to connect to itself via address configured in advertised.listeners ?
2. Is there a way to disable this or at least configure it to use address from 'listeners' property (with default Kafka port) ?

UPD Worth to mention, following setup does not work: Kafka listens on 0.0.0.0:9092, advertised listener is configured to docker-host:4005.
In this case whenever consumer or producer connects to kafka it receives LEADER_NOT_AVAILABLE. There is also connection shown by netstat (within container) to docker-host:4005 in state SYN_SENT.

UPD 2 Looks like there is similar problem with Kafka but inside AWS described here.
Difference is that in my case I want to use different Kafka port.

UPD 3 Ok, the reason why setup mentioned in the first UPD paragraph does not work is - UFW, for some reasons it blocks traffic which goes from docker container to itself via host machine.

Answer 1

Why Kafka broker might need to connect to itself via address configured in advertised.listeners ?

When a Kafka broker is first connected by a client, it replies back with the address that it expects that client to use in the future to talk to the broker. This is what is set in the advertised.listeners property. If you don't set this property, the value from listeners will be used instead (which answers your second question).

So your "issue" is, that remote clients connect to yourhost:9092, reach the Kafka broker, because you forwarded the port, the broker then responds with "you can reach me at localhost:9092" and when the client sends the next packet there it just connects back to itself. The metadata is not really related here, its just the first request that gets made. Your solution is correct for this setup I think, have Kafka listen on local interfaces and set the advertised.listeners to the host that someone from your company network would connect to. I don't 100% know if the broker needs to connect to itself as well, pretty sure thats not the case though. I think your setup would also work without the entry of the external hostname in your /etc/hosts file.

Is there a way to disable this or at least configure it to use address from 'listeners' property (with default Kafka port) ?

see above