How to fix Kafka Docker container from throwing 0.0.0.0/0.0.0.0:2181: Connection refused?

Question

I am trying to set up a Docker compose file environment for Kafka change data capture and I am encountering this error:

Opening socket connection to server 0.0.0.0/0.0.0.0:2181. Will not attempt to authenticate using SASL (unknown error)

Socket error occurred: 0.0.0.0/0.0.0.0:2181: Connection refused

I have been following this tutorial https://hevodata.com/learn/kafka-cdc-postgres/, but it is running the docker commands directly using the link option and not using a docker-compose.yml file.

I attempted to convert these:

docker run -it --name kafka -p 9092:9092 --link zookeeper:zookeeper debezium/kafka

to the below docker-compose.yml file. However, it appears it is completely ignoring the KAFKA_ZOOKEEPER_CONNECT environment variable as this is what I see in the log:

Using ZOOKEEPER_CONNECT=0.0.0.0:2181

Even though, the documentation https://github.com/debezium/docker-images/tree/master/kafka/1.5 indicates it should work.

When I follow the tutorial using docker run and not creating a docker-compose file it works completely fine. It shows my local computer's IP address with port 2181 instead of 0.0.0.0:2181.

docker-compose.yml:

version: "3.7"

services:
  postgres:
    image: debezium/postgres:10
    container_name: postgres
    ports:
      - "5000:5432"
    environment:
      POSTGRES_HOST_AUTH_METHOD: trust
      # POSTGRES_USER: db_user
      # POSTGRES_PASSWORD: db_password
  zookeeper:
    image: debezium/zookeeper:1.5
    container_name: zookeeper
    ports:
      - "2181:2181"
      - "2888:2888"
      - "3888:3888"
  kafka:
    image: debezium/kafka:1.5
    container_name: kafka
    ports:
      - "9092:9092"
    depends_on:
      - zookeeper
    environment:
      KAFKA_ADVERTISED_HOST_NAME: kafka
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
  connect:
    image: debezium/connect:1.5
    container_name: connect
    ports:
      - "8083:8083"
    environment:
      GROUP_ID: 1
      CONFIG_STORAGE_TOPIC: my-connect-configs
      OFFSET_STORAGE_TOPIC: my-connect-offsets
    depends_on: 
      - postgres
      - kafka
      - zookeeper

networks:
  default:
    name: kafkaCDC

zoo.cfg on the Zookeeper container:

tickTime=2000
initLimit=10
syncLimit=5
dataDir=/zookeeper/data
dataLogDir=/zookeeper/txns
clientPort=2181
autopurge.snapRetainCount=3
autopurge.purgeInterval=1

Been looking at this issue for several; however, I am getting totally lost. Especially, since so many examples are using links.

This is the GitHub post that made me think to use KAFKA_ZOOKEEPER_CONNECT. https://github.com/wurstmeister/kafka-docker/issues/512#issuecomment-505905161

Apart of me feels like it is something wrong with https://github.com/debezium/docker-images/blob/master/kafka/1.5/docker-entrypoint.sh that is ignoring the environment variable, but it is probably just me not understanding something and having a conf error.

Answer 1

Don't prefix your ENVIRONMENT variables with KAFKA_

Here is my working cluster :

version: '2'
services:
  postgres:
    image: debezium/postgres:13-alpine
    container_name: postgres
    hostname: postgres
    environment:
      POSTGRES_USER: nikamooz
      POSTGRES_PASSWORD: nikamooz
      
    ports:
      - 5432:5432
    
  zookeeper:
    image: debezium/zookeeper
    container_name: zookeeper
    hostname: zookeeper
    environment:
      ZOOKEEPER_SERVER_ID: 1
    ports:
      - 2182:2181
      - 2888:2888
      - 3888:3888
    volumes: 
      - ./data/zoo/data:/zookeeper/data
      - ./data/zoo/log:/zookeeper/txns

  kafka:
    image: debezium/kafka
    container_name: kafka
    hostname: kafka
    depends_on:
      - zookeeper
    ports:
      - 9092:9092
    environment:
      ZOOKEEPER_CONNECT: zookeeper:2181
      BOOTSTRAP_SERVERS: kafka:9092
    volumes: 
      - ./data/kafka/data:/kafka/data
      - ./data/kafka/logs:/kafka/logs


  connect:
    image: debezium/connect
    container_name: connect
    hostname: connect
    depends_on:
      - kafka
      - postgres
    ports:
      - 8083:8083
    environment:
      GROUP_ID: holding_group
      CONFIG_STORAGE_TOPIC: holding_storage_topic
      OFFSET_STORAGE_TOPIC: holding_offset_topic
      BOOTSTRAP_SERVERS: kafka:9092
     

Answer 2

For debezium/kafka:1.5 image to work in docker compose, you can try passing following environment variable:

ZOOKEEPER_CONNECT: "zookeeper:2181"

It addressed my problem, sample docker compose yaml below:

version: "3.9"
services:
  zookeeper:
    image: debezium/zookeeper:1.5
    ports:
      - "2181:2181"
      - "2888:2888"
  kafka:
    image: debezium/kafka:1.5
    ports:
      - "9092:9092"
    environment:
      ZOOKEEPER_CONNECT: "zookeeper:2181"
    depends_on:
      - zookeeper

Answer 3

I was able to fix it by setting the zookeeper connect address to the docker container IP address

To get the IP address run

docker inspect <container-name> --format='{{ .NetworkSettings.IPAddress }}'

and start start kafka as follows

docker run --name some-kafka -p 9092:9092 -e KAFKA_ZOOKEEPER_CONNECT=<zookeeper-ip>:2181 -e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://:9092 -e KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1 confluentinc/cp-kafka

This github comment helped me find out what I was missing