kafka-node can't connect to kafka server in local environment

Question

I started up a kafka (version 2.2.0) docker container, by using the docker inspect command.

kafka config is

Using ZOOKEEPER_CONNECT=172.17.0.2:2181
Using KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://172.17.0.6:9092
Using KAFKA_BROKER=172.17.0.4:9092

The network setting is shown

 "NetworkSettings": {
            "Ports": {
                "8778/tcp": null,
                "9092/tcp": null,
                "9779/tcp": null
            },
            "Gateway": "172.17.0.1",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "172.17.0.6",
            "IPPrefixLen": 16,
            "IPv6Gateway": "",
            "MacAddress": "02:42:ac:11:00:06"

Also I looked up the topics in the kafka container.

./zkCli.sh
ls /brokers/topics

topics are shown below

[connect-status, dbserver1.public.dumb_table, my-connect-offsets, __consumer_offsets, my-connect-configs]

At last I started a node app to connect to kafka but failed. (https://github.com/SOHU-Co/kafka-node#kafkaclient)

const kafka = require('kafka-node');
const bp = require('body-parser');

try {
  const Consumer = kafka.HighLevelConsumer;
  const client = new kafka.KafkaClient({kafkaHost: '172.17.0.6:9092'});
  let consumer = new kafka.Consumer(
    client,
    [{ topic: "dbserver1.public.dumb_table", partition: 0 }],
    {
      autoCommit: true,
      fetchMaxWaitMs: 1000,
      fetchMaxBytes: 1024 * 1024,
      encoding: 'utf8',
      fromOffset: false
    }
  );
  consumer.on('ready', function () {
        console.log('consumer ready');

  });
  consumer.on('message', async function(message) {
    console.log('here');
    console.log(
      'kafka-> ',
      message.value
    );
  })
  consumer.on('error', function(err) {
    console.log('error', err);
  });
}
catch(e) {
  console.log(e);
}

if kafkaHost is configured as '172.17.0.6:9092', an error is threw out

{ Error: connect ENETUNREACH 172.17.0.6:9092
    at Object._errnoException (util.js:992:11)
    at _exceptionWithHostPort (util.js:1014:20)
    at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1186:14)
  code: 'ENETUNREACH',
  errno: 'ENETUNREACH',
  syscall: 'connect',
  address: '172.17.0.6',
  port: 9092 }

if kafkaHost is configured as 'localhost:9092', an error is threw out

{ TimeoutError: Request timed out after 30000ms
at new TimeoutError (/Users/xisizhe/Documents/projects/game/server/node_modules/kafka-node/lib/errors/TimeoutError.js:6:9)
at Timeout.setTimeout [as _onTimeout] (/Users/xisizhe/Documents/projects/game/server/node_modules/kafka-node/lib/kafkaClient.js:491:14)
at ontimeout (timers.js:498:11)
at tryOnTimeout (timers.js:323:5)
at Timer.listOnTimeout (timers.js:290:5) message: 'Request timed out after 30000ms' }

Answer 1

if kafkaHost is configured as '172.17.0.6:9092', an error is threw out

Is your Node app running in Docker as well? If so, you would not see that error.

---

It looks like you did docker inspect, then copied some IPs out of that, which I don't see that being required in any Debezium / Docker+Kafka tutorial I've come across.

You want the host IP (your Mac); outside of Docker and the VM network it is hosted within (which your host cannot route requests to that subnet).

---

e.g. On my Mac, I inspect a random container (which just happens to be Kafka)

docker inspect ee708cf11010 | grep IPAddress
        "SecondaryIPAddresses": null,
        "IPAddress": "",
                "IPAddress": "172.18.0.3",
                "IPAddress": "172.20.0.3",

I cannot connect to those

nc -vz 172.20.0.3 9092
# ... hangs

But I can connect to the exposed port just fine

nc -vz localhost 9092
found 0 associations
found 1 connections:
     1: flags=82<CONNECTED,PREFERRED>
    outif lo0
    src ::1 port 51180
    dst ::1 port 9092
    rank info not available
    TCP aux info available

Connection to localhost port 9092 [tcp/XmlIpcRegSvc] succeeded!

Or

• using my hostname with nc -vz $(hostname -f) 9092
• my LAN IP nc -vz 192.168.1.105 9092

---

As the Debezium tutorial states, ADVERTISED_HOST_NAME needs to be your host's hostname or external IP. The hostname is simplest, and you can pass that to docker like so.

docker run -it --name kafka -p 9092:9092 -e ADVERTISED_HOST_NAME=$(hostname -f) --link zookeeper:zookeeper debezium/kafka

---

Your Node code can still use localhost:9092, but it would be more logical to pass the proper IP/hostname anyway.

---

Aside: you are not required to use Debezium/kafka container to use the actual Debezium connectors since its just starting Kafka. So, here's my other answer on how you'd properly configure Kafka in Docker for producer/consumer apps running both in Docker and from your host

Connect to Kafka running in Docker from local machine