CODEWITHSUNDEEP
apache-kafkaapache-zookeeperdocker-swarm
ugur
ugur

Reputation: 410

Apache Kafka and Zookeeper connection problem

I am trying to create a multi-node Apache Kafka cluster by using Docker swarm. To do that, I use the following docker-compose.yml 

version: '3.2'
services:
   zookeeper:
      image: wurstmeister/zookeeper
      ports:
         - "2181:2181"

   kafka:
      image: wurstmeister/kafka:latest
      deploy:
         mode: global
      ports:
         - target: 9094
           published: 9094
           protocol: tcp
           mode: host
      environment:
         HOSTNAME_COMMAND: "docker info | grep ^Name: | cut -d' ' -f 2" # Normal instances
         KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
         KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
         KAFKA_ADVERTISED_PROTOCOL_NAME: OUTSIDE
         KAFKA_ADVERTISED_PORT: 9094
         KAFKA_PROTOCOL_NAME: INSIDE
         KAFKA_PORT: 9092
         KAFKA_CREATE_TOPICS: myTopic:1:1
      volumes:
         - /var/run/docker.sock:/var/run/docker.sock

But, when kafka_kafka service is created, I encounter with following error:

java.lang.IllegalArgumentException: Error creating broker listeners from 'PLAINTEXT://ugur:9094': No security protocol defined for listener PLAINTEXT
kafka_kafka.0.wx4chyussuuc@ugur    |    at kafka.utils.CoreUtils$.listenerListToEndPoints(CoreUtils.scala:312)
kafka_kafka.0.wx4chyussuuc@ugur    |    at kafka.server.KafkaConfig.advertisedListeners(KafkaConfig.scala:1300)
kafka_kafka.0.wx4chyussuuc@ugur    |    at kafka.server.KafkaConfig.validateValues(KafkaConfig.scala:1360)
kafka_kafka.0.wx4chyussuuc@ugur    |    at kafka.server.KafkaConfig.<init>(KafkaConfig.scala:1338)
kafka_kafka.0.wx4chyussuuc@ugur    |    at kafka.server.KafkaConfig.<init>(KafkaConfig.scala:1028)
kafka_kafka.0.wx4chyussuuc@ugur    |    at kafka.server.KafkaConfig$.fromProps(KafkaConfig.scala:1008)
kafka_kafka.0.wx4chyussuuc@ugur    |    at kafka.server.KafkaServerStartable$.fromProps(KafkaServerStartable.scala:28)
kafka_kafka.0.wx4chyussuuc@ugur    |    at kafka.Kafka$.main(Kafka.scala:59)
kafka_kafka.0.wx4chyussuuc@ugur    |    at kafka.Kafka.main(Kafka.scala)
kafka_kafka.0.wx4chyussuuc@ugur    | Caused by: java.lang.IllegalArgumentException: No security protocol defined for listener PLAINTEXT
kafka_kafka.0.wx4chyussuuc@ugur    |    at kafka.cluster.EndPoint$$anonfun$securityProtocol$1$1.apply(EndPoint.scala:48)
kafka_kafka.0.wx4chyussuuc@ugur    |    at kafka.cluster.EndPoint$$anonfun$securityProtocol$1$1.apply(EndPoint.scala:48)
kafka_kafka.0.wx4chyussuuc@ugur    |    at scala.collection.MapLike$class.getOrElse(MapLike.scala:128)
kafka_kafka.0.wx4chyussuuc@ugur    |    at scala.collection.AbstractMap.getOrElse(Map.scala:59)
kafka_kafka.0.wx4chyussuuc@ugur    |    at kafka.cluster.EndPoint$.securityProtocol$1(EndPoint.scala:47)
kafka_kafka.0.wx4chyussuuc@ugur    |    at kafka.cluster.EndPoint$.createEndPoint(EndPoint.scala:56)
kafka_kafka.0.wx4chyussuuc@ugur    |    at kafka.utils.CoreUtils$$anonfun$6.apply(CoreUtils.scala:309)
kafka_kafka.0.wx4chyussuuc@ugur    |    at kafka.utils.CoreUtils$$anonfun$6.apply(CoreUtils.scala:309)
kafka_kafka.0.wx4chyussuuc@ugur    |    at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
kafka_kafka.0.wx4chyussuuc@ugur    |    at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
kafka_kafka.0.wx4chyussuuc@ugur    |    at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
kafka_kafka.0.wx4chyussuuc@ugur    |    at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:35)
kafka_kafka.0.wx4chyussuuc@ugur    |    at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
kafka_kafka.0.wx4chyussuuc@ugur    |    at scala.collection.AbstractTraversable.map(Traversable.scala:104)
kafka_kafka.0.wx4chyussuuc@ugur    |    at kafka.utils.CoreUtils$.listenerListToEndPoints(CoreUtils.scala:309)

I also open necessary ports, and these ports like that:

To Action From

-- ------ ----

9094 ALLOW IN Anywhere
9092 ALLOW IN Anywhere
8081 ALLOW IN Anywhere
8080 ALLOW IN Anywhere
2377 ALLOW IN Anywhere
7946 ALLOW IN Anywhere
4789 ALLOW IN Anywhere
9094 (v6) ALLOW IN Anywhere (v6)
9092 (v6) ALLOW IN Anywhere (v6)
8081 (v6) ALLOW IN Anywhere (v6)
8080 (v6) ALLOW IN Anywhere (v6)
2377 (v6) ALLOW IN Anywhere (v6)
7946 (v6) ALLOW IN Anywhere (v6)
4789 (v6) ALLOW IN Anywhere (v6)

Upvotes: 0

Views: 5178

Answers (2)

jido
jido

Reputation: 1

I have added the following options:

  KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT
  KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
  KAFKA_LISTENERS: INTERNAL://:29092,EXTERNAL://:9092
  KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka:29092,EXTERNAL://localhost:9092
  ALLOW_PLAINTEXT_LISTENER: "yes"

Obviously plaintext listener is not secure but it should get you started.

Upvotes: 0

Marcelo Rubim
Marcelo Rubim

Reputation: 137

Did you try using using the docker-compose-swarm.yml available at GitHub?

https://github.com/wurstmeister/kafka-docker/blob/master/docker-compose-swarm.yml

Maybe the missing property bellow is the key:

KAFKA_ADVERTISED_LISTENERS: INSIDE://:9092,OUTSIDE://_{HOSTNAME_COMMAND}:9094

Upvotes: 2

Related Questions