CODEWITHSUNDEEP
dockerapache-kafkadocker-compose
ISM
ISM

Reputation: 61

Docker Error : kafka.zookeeper.ZooKeeperClientTimeoutException in Kafka shell

I'm trying to run kafka using docker.

I started kafka service.

docker-compose ps

     Name                        Command               State                         Ports

kafka-docker_kafka_1 start-kafka.sh Up 0.0.0.0:9092->9092/tcp
kafka-docker_zookeeper_1 /bin/sh -c /usr/sbin/sshd ... Up 0.0.0.0:2181->2181/tcp, 22/tcp, 2888/tcp, 3888/tcp

docker-compose.yml

cat docker-compose.yml

version: '2'
services:
  zookeeper:
    image: wurstmeister/zookeeper
    ports:
      - "2181:2181"
  kafka:
    build: .
    ports:
      - "9092:9092"
    environment:
      KAFKA_ADVERTISED_HOST_NAME: kafka
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ZOOKEEPER_CONNECTION_TIMEOUT_MS: 36000
      KAFKA_ADVERTISED_PORT: 9092
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

When I tried to create a topic by entering inside shell, it's throwing the below error.

./start-kafka-shell.sh kafka zookeeper:2181

bash-4.4# $KAFKA_HOME/bin/kafka-topics.sh --create --topic sentiment --partitions 1 --zookeeper $ZK --replication-factor 1

[2019-02-12 08:07:59,097] WARN Session 0x0 for server zookeeper:2181, unexpected error, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
java.nio.channels.UnresolvedAddressException
    at sun.nio.ch.Net.checkAddress(Net.java:101)
    at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:622)
    at org.apache.zookeeper.ClientCnxnSocketNIO.registerAndConnect(ClientCnxnSocketNIO.java:277)
    at org.apache.zookeeper.ClientCnxnSocketNIO.connect(ClientCnxnSocketNIO.java:287)
    at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1021)
    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1064)

Tried 2 possibilities:

  1. Got timeout exception on changing hostname :zookeeper(though I mentioned the same in docker-compose.yml ) to ip: 0.0.0.0

./start-kafka-shell.sh kafka 0.0.0.0:2181

bash-4.4# $KAFKA_HOME/bin/kafka-topics.sh --create --topic sentiment --partitions 1 --zookeeper $ZK --replication-factor 1

Exception in thread "main" kafka.zookeeper.ZooKeeperClientTimeoutException: Timed out waiting for connection while in state: CONNECTING
    at kafka.zookeeper.ZooKeeperClient.$anonfun$waitUntilConnected$3(ZooKeeperClient.scala:268)
    at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
    at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:251)
    at kafka.zookeeper.ZooKeeperClient.waitUntilConnected(ZooKeeperClient.scala:264)
    at kafka.zookeeper.ZooKeeperClient.<init>(ZooKeeperClient.scala:97)
    at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:1694)
    at kafka.admin.TopicCommand$.main(TopicCommand.scala:57)
    at kafka.admin.TopicCommand.main(TopicCommand.scala)
  1. Inorder to avoid the above error, I increased the timeout parameter from 6000 to 36000 in docker-compose.yml --> KAFKA_ZOOKEEPER_CONNECTION_TIMEOUT_MS: 36000

I'm getting the same timeout error.

Upvotes: 3

Views: 5672

Answers (3)

dev Joshi
dev Joshi

Reputation: 415

In my case windows firewall was blocking docker containers. I ran below in powershell to resolve Set-NetConnectionProfile -interfacealias "vEthernet (DockerNAT)" -NetworkCategory Private

Upvotes: 0

Nikolay Baranenko
Nikolay Baranenko

Reputation: 1675

yesterday I started use kafka in docker from https://github.com/wurstmeister/kafka-docker.git

my steps

https://github.com/wurstmeister/kafka-docker.git

nano kafka-docker/docker-compose.yml

my conf

version: '3'
services:
  zookeeper:
    image: wurstmeister/zookeeper
    ports:
      - "2181:2181"
  kafka:
    image: wurstmeister/kafka
    ports:
      - "9092"
    environment:
      KAFKA_ADVERTISED_HOST_NAME: localhost
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_CREATE_TOPICS: "test-topic:5:2"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

after 

docker-compose up -d 

docker-compose scale kafka=3

check docker container

$docker-compose ps
          Name                        Command               State                         Ports                       
----------------------------------------------------------------------------------------------------------------------
kafka-docker_kafka_1       start-kafka.sh                   Up      0.0.0.0:32784->9092/tcp                           
kafka-docker_kafka_2       start-kafka.sh                   Up      0.0.0.0:32785->9092/tcp                           
kafka-docker_kafka_3       start-kafka.sh                   Up      0.0.0.0:32786->9092/tcp                           
kafka-docker_zookeeper_1   /bin/sh -c /usr/sbin/sshd  ...   Up      0.0.0.0:2181->2181/tcp, 22/tcp, 2888/tcp, 3888/tcp

or

$ docker ps -a
CONTAINER ID        IMAGE                    COMMAND                  CREATED             STATUS              PORTS                                                NAMES
1479a7dc96fa        wurstmeister/kafka       "start-kafka.sh"         About an hour ago   Up About an hour    0.0.0.0:32785->9092/tcp                              kafka-docker_kafka_2
08e5017dae2b        wurstmeister/kafka       "start-kafka.sh"         About an hour ago   Up About an hour    0.0.0.0:32786->9092/tcp                              kafka-docker_kafka_3
d4f3d17e81b2        wurstmeister/kafka       "start-kafka.sh"         About an hour ago   Up About an hour    0.0.0.0:32784->9092/tcp                              kafka-docker_kafka_1
72b0fbe553b5        wurstmeister/zookeeper   "/bin/sh -c '/usr/sb…"   About an hour ago   Up About an hour    22/tcp, 2888/tcp, 3888/tcp, 0.0.0.0:2181->2181/tcp   kafka-docker_zookeeper_1

after connect to kafka broker1 and get topics from zookeeper

$ docker exec -it kafka-docker_kafka_1 bash
bash-4.4# /opt/kafka/bin/kafka-topics.sh --zookeeper zookeeper:2181 --list
sample
test-topic
bash-4.4#

get info by topics

bash-4.4# /opt/kafka/bin/kafka-topics.sh --zookeeper zookeeper:2181 --describe --topic sample
Topic: sample   PartitionCount: 1   ReplicationFactor: 1    Configs: 
    Topic: sample   Partition: 0    Leader: 1002    Replicas: 1002  Isr: 1002
bash-4.4# /opt/kafka/bin/kafka-topics.sh --zookeeper zookeeper:2181 --describe --topic test-topic
Topic: test-topic   PartitionCount: 5   ReplicationFactor: 2    Configs: 
    Topic: test-topic   Partition: 0    Leader: 1001    Replicas: 1001,1003 Isr: 1001,1003
    Topic: test-topic   Partition: 1    Leader: 1002    Replicas: 1002,1001 Isr: 1002,1001
    Topic: test-topic   Partition: 2    Leader: 1003    Replicas: 1003,1002 Isr: 1003,1002
    Topic: test-topic   Partition: 3    Leader: 1001    Replicas: 1001,1002 Isr: 1001,1002
    Topic: test-topic   Partition: 4    Leader: 1002    Replicas: 1002,1003 Isr: 1002,1003

after create new topic

bash-4.4# /opt/kafka/bin/kafka-topics.sh --zookeeper zookeeper:2181 --create --topic topic1 --replication-factor 2 --partitions 2
Created topic topic1.

describe

bash-4.4# /opt/kafka/bin/kafka-topics.sh --zookeeper zookeeper:2181 --describe --topic topic1
Topic: topic1   PartitionCount: 2   ReplicationFactor: 2    Configs: 
    Topic: topic1   Partition: 0    Leader: 1003    Replicas: 1003,1001 Isr: 1003,1001
    Topic: topic1   Partition: 1    Leader: 1001    Replicas: 1001,1002 Isr: 1001,1002

Upvotes: 1

OneCricketeer
OneCricketeer

Reputation: 192023

There are lots of pre-built Kafka images. It's hard to tell what you're actually trying to do, but seems you're rebuilding an image? Or building your own?

You definitely don't need the zookeeper address as a parameter to start Kafka, though

That being said, 0.0.0.0 is not a real address to connect to. It's often used as a bind address meaning to "allow all connections to this server"

Then zookeeper service name is only resolvable within the Docker network, so that might explain why you need --net, but that implies you're using docker run for something, which again isn't clear given you're using Compose

If you want to get into a shell of the container, you should just be to use docker-compose exec kafka bash, and Zookeeper is already an environment variable of that container

So something like

docker-compose exec kafka bash -c '$KAFKA_HOME/bin/kafka-topics.sh --create --topic sentiment --partitions 1 --zookeeper $KAFKA_ZOOKEEPER_CONNECT --replication-factor 1'

Upvotes: 3

Related Questions