CODEWITHSUNDEEP
apache-kafkadocker-swarmdebezium
Hiếu Trần
Hiếu Trần

Reputation: 23

Debezium error when connecting to kafka multi-broker in docker swarm

when I set up my Swarm with this stack: Kafka(multi-broker), zookeeper, debezium. Kafka and zookeeper are working, can create topic, consumer and producer, but debezium show error: org.apache.kafka.connect.errors.ConnectException: Failed to connect to and describe Kafka cluster. Check worker's broker connection and security properties. I'm not modifying anything, just default config as docker-stack below:

version: '3.6'
services:
   zoo:
      image: wurstmeister/zookeeper
      ports:
         - '2181:2181'
      volumes:
         - zoo-data:/tmp/zookeeper
      deploy:
         replicas: 1
         placement:
            constraints:
               - node.labels.type==zoo
   kafka:
      image: wurstmeister/kafka:latest
      ports:
         - target: 9094
           published: 9094
           protocol: tcp
           mode: host
      environment:
         HOSTNAME_COMMAND: "docker info | grep ^Name: | cut -d' ' -f 2"
         KAFKA_ZOOKEEPER_CONNECT: zoo:2181
         KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
         KAFKA_ADVERTISED_LISTENERS: INSIDE://:9092,OUTSIDE://_{HOSTNAME_COMMAND}:9094
         KAFKA_LISTENERS: INSIDE://:9092,OUTSIDE://:9094
         KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE
         #KAFKA_CREATE_TOPICS: "Topic1:1:2,Topic2:1:1:compact"
      volumes:
         - /var/run/docker.sock:/var/run/docker.sock
         - kafka-data:/tmp/kafka-logs
      deploy:
         mode: global
         placement:
            constraints:
               - node.labels.name==kafka
      depends_on:
         - zoo

   debezium:
      image: debezium/connect:0.8
      hostname: connect
      ports:
         - '8083:8083'
      environment:
         BOOTSTRAP_SERVERS: kafka:9094
         GROUP_ID: 1
         CONFIG_STORAGE_TOPIC: my_connect_configs
         OFFSET_STORAGE_TOPIC: my_connect_offsets
      deploy:
         placement:
            constraints:
               - node.labels.type==dbz
      depends_on:
         - kafka
volumes:
   kafka-data:
   zoo-data:

When I check docker services log debezium, it show error

shippo_kafka_debezium.1.5l1yhz27r6p2@kafka1    | 2018-08-03 04:33:27,034 ERROR  ||  Stopping due to error   [org.apache.kafka.connect.cli.ConnectDistributed]
shippo_kafka_debezium.1.5l1yhz27r6p2@kafka1    | org.apache.kafka.connect.errors.ConnectException: Failed to connect to and describe Kafka cluster. Check worker's broker connection and security properties.
shippo_kafka_debezium.1.5l1yhz27r6p2@kafka1    |    at org.apache.kafka.connect.util.ConnectUtils.lookupKafkaClusterId(ConnectUtils.java:64)
shippo_kafka_debezium.1.5l1yhz27r6p2@kafka1    |    at org.apache.kafka.connect.util.ConnectUtils.lookupKafkaClusterId(ConnectUtils.java:45)
shippo_kafka_debezium.1.5l1yhz27r6p2@kafka1    |    at org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:77)
shippo_kafka_debezium.1.5l1yhz27r6p2@kafka1    | Caused by: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Timed out waiting to send the call.
shippo_kafka_debezium.1.5l1yhz27r6p2@kafka1    |    at org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
shippo_kafka_debezium.1.5l1yhz27r6p2@kafka1    |    at org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
shippo_kafka_debezium.1.5l1yhz27r6p2@kafka1    |    at org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89)
shippo_kafka_debezium.1.5l1yhz27r6p2@kafka1    |    at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:258)
shippo_kafka_debezium.1.5l1yhz27r6p2@kafka1    |    at org.apache.kafka.connect.util.ConnectUtils.lookupKafkaClusterId(ConnectUtils.java:58)
shippo_kafka_debezium.1.5l1yhz27r6p2@kafka1    |    ... 2 more
shippo_kafka_debezium.1.5l1yhz27r6p2@kafka1    | Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting to send the call.

Can anyone show me how to fix this error, I'm new with this stack so, during a few days research, I can't figure out it. Thank you so much !

Upvotes: 0

Views: 1077

Answers (1)

Roham Mosalli
Roham Mosalli

Reputation: 11

You can change this line HOSTNAME_COMMAND: "docker info | grep ^Name: | cut -d' ' -f 2" to HOSTNAME_COMMAND: "docker info | grep 'Node Address:' | cut -d' ' -f 4"

or just you can use this docker-compose file 

version: '3.2'
services:
  zookeeper:
    image: wurstmeister/zookeeper
    ports:
      - "2181:2181"
    deploy:
       mode: global
    volumes:
      - /shared_data/zoo1/data:/data
      - /shared_data/zoo1/datalog:/datalog
    environment:
        ZOO_MY_ID: 1
        ZOO_PORT: 2181
        ZOO_SERVERS: server.1=zookeeper:2888:3888

  kafka:
    image: wurstmeister/kafka:latest
    ports:
      - target: 9094
        published: 9094
        protocol: tcp
        mode: host
    deploy:
      mode: global
    environment:
      HOSTNAME_COMMAND: "docker info | grep 'Node Address:' | cut -d' ' -f 4"
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: INSIDE://:9092,OUTSIDE://_{HOSTNAME_COMMAND}:9094
      KAFKA_LISTENERS: INSIDE://:9092,OUTSIDE://:9094
      KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - /shared_data/kafka:/var/lib/kafka/data
    depends_on:
      - zookeeper

Upvotes: 1

Related Questions