CODEWITHSUNDEEP
apache-kafkadocker-composeapache-kafka-connect
user824624
user824624

Reputation: 8080

Broker may not be available in connecting kafka connector with kafka broker in docker compose

I am using docker container to build zookeeper, kafka and connect.

version: '2.1'
services:


  zookeeper:
    image: debezium/zookeeper
    ports:
     - 2181:2181
     - 2888:2888
     - 3888:3888
  kafka:
    image: wurstmeister/kafka
    container_name: kafka-multibinder-1
    ports:
      - "9092:9092"
      - "9094:9094"

    environment:
      - KAFKA_ADVERTISED_HOST_NAME=127.0.0.1
      - KAFKA_ADVERTISED_PORT=9092
      - KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181

      - KAFKA_ADVERTISED_LISTENERS=INSIDE://:9094,OUTSIDE://localhost:9092
      - KAFKA_LISTENERS=INSIDE://:9094,OUTSIDE://:9092
      - KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
      - KAFKA_INTER_BROKER_LISTENER_NAME=INSIDE

    depends_on:
      - zookeeper

  kafka-connect:
    image: debezium/connect
    hostname: kafka-connect
    ports:
     - 8083:8083
    depends_on:
     - kafka
    environment:
     BOOTSTRAP_SERVERS: kafka:9092
     GROUP_ID: 1
     CONFIG_STORAGE_TOPIC: my_connect_configs
     OFFSET_STORAGE_TOPIC: my_connect_offsets

however the logs warms me that 'could not be established. Broker may not be available'

kafka-connect_1  | 2020-10-18 04:11:44,671 INFO   ||  Kafka version: 2.5.0   [org.apache.kafka.common.utils.AppInfoParser]
kafka-connect_1  | 2020-10-18 04:11:44,672 INFO   ||  Kafka commitId: 66563e712b0b9f84   [org.apache.kafka.common.utils.AppInfoParser]
kafka-connect_1  | 2020-10-18 04:11:44,674 INFO   ||  Kafka startTimeMs: 1602994304669   [org.apache.kafka.common.utils.AppInfoParser]
kafka-connect_1  | 2020-10-18 04:11:44,806 WARN   ||  [AdminClient clientId=adminclient-1] Connection to node -1 (/127.0.0.1:9092) could not be established. Broker may not be available.   [org.apache.kafka.clients.NetworkClient]
kafka-connect_1  | 2020-10-18 04:11:44,918 WARN   ||  [AdminClient clientId=adminclient-1] Connection to node -1 (/127.0.0.1:9092) could not be established. Broker may not be available.   [org.apache.kafka.clients.NetworkClient]
kafka-connect_1  | 2020-10-18 04:11:45,024 WARN   ||  [AdminClient clientId=adminclient-1] Connection to node -1 (/127.0.0.1:9092) could not be established. Broker may not be available.   [org.apache.kafka.clients.NetworkClient]
kafka-connect_1  | 2020-10-18 04:12:44,713 INFO   ||  [AdminClient clientId=adminclient-1] Metadata update failed   [org.apache.kafka.clients.admin.internals.AdminMetadataManager]
kafka-connect_1  | org.apache.kafka.common.errors.TimeoutException: Call(callName=fetchMetadata, deadlineMs=1602994364719) timed out at 9223372036854775807 after 1 attempt(s)
kafka-connect_1  | Caused by: org.apache.kafka.common.errors.TimeoutException: The AdminClient thread has exited.
kafka-connect_1  | 2020-10-18 04:12:44,722 ERROR  ||  Stopping due to error   [org.apache.kafka.connect.cli.ConnectDistributed]
kafka-connect_1  | org.apache.kafka.connect.errors.ConnectException: Failed to connect to and describe Kafka cluster. Check worker's broker connection and security properties.
kafka-connect_1  |  at org.apache.kafka.connect.util.ConnectUtils.lookupKafkaClusterId(ConnectUtils.java:64)
kafka-connect_1  |  at org.apache.kafka.connect.util.ConnectUtils.lookupKafkaClusterId(ConnectUtils.java:45)
kafka-connect_1  |  at org.apache.kafka.connect.cli.ConnectDistributed.startConnect(ConnectDistributed.java:95)
kafka-connect_1  |  at org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:78)
kafka-connect_1  | Caused by: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Call(callName=listNodes, deadlineMs=1602994364707) timed out at 1602994364708 after 1 attempt(s)
kafka-connect_1  |  at org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
kafka-connect_1  |  at org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
kafka-connect_1  |  at org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89)
kafka-connect_1  |  at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:260)
kafka-connect_1  |  at org.apache.kafka.connect.util.ConnectUtils.lookupKafkaClusterId(ConnectUtils.java:58)
kafka-connect_1  |  ... 3 more
kafka-connect_1  | Caused by: org.apache.kafka.common.errors.TimeoutException: Call(callName=listNodes, deadlineMs=1602994364707) timed out at 1602994364708 after 1 attempt(s)
kafka-connect_1  | Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment.

Upvotes: 1

Views: 7556

Answers (2)

Lisen Saka
Lisen Saka

Reputation: 56

I was facing the same issue while trying to set up Kafka through a docker compose file and after spending few research hours i solved it like this:

on the Kafka service at docker file, i defined a configuration field called KAFKA_ADVERTISED_HOST_NAME, with this config we define the address/IP that clients (producers and consumers) should use in order to connect with the Kafka brokers, so i configured it like this:

      KAFKA_ADVERTISED_HOST_NAME: localhost

and then when i started again the docker compose file the clients were able to connect to the broker, you can see the Kafka logs that says the message highlighted on the screenshot below: enter image description here

After that i configured in my spring boot application, on the application.propertied file, i defined the field:

spring.kafka.bootstrap-servers=localhost:9092

Upvotes: 1

Ravi
Ravi

Reputation: 108

In your Kafka configuration port 9094 is used for inter broker communication. But in Kafka connect you trying to connect to the broker via port 9092.

As you are trying to connect to port 9092 then localhost:9092 will be sent as metadata to the Kafka connect for further communication but localhost:9092 is not available inside the docker network. Use below Kafka connect configuration to connect -

kafka-connect:
    image: debezium/connect
    hostname: kafka-connect
    ports:
     - 8083:8083
    depends_on:
     - kafka
    environment:
     BOOTSTRAP_SERVERS: kafka:9094
     GROUP_ID: 1
     CONFIG_STORAGE_TOPIC: my_connect_configs
     OFFSET_STORAGE_TOPIC: my_connect_offsets

The concept of Advertised listeners is a little bit trickiest. Refer to this link if you want to know more about listeners and advertised listeners.

Upvotes: 3

Related Questions