kafka bootstrap broker disconnected using docker-compose

Question

Good morning. First of all, I've created a docker swarm with 2 physical hosts and an overlay network. In the same host I've created 2 containers (postgres and ambari servers) and one with ambari agent in which I'll install kafka, zookeeper, spark,... from ambari. The scope is installing several containers in several hosts, but I'm trying first like this as I'm not getting it working.

The fact is that once deployed with Ambari, I change kafka configuration to add advertised.host.name to the physical host's ip and advertised.port to 9092 to bind it to physical host's 9092 port.

When trying it, I'm always getting the following errors:

WARN Error while fetching metadata with correlation id 17 : {prueba2=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)

If trying by sending to container's 6667 port or

[2018-08-30 12:06:28,758] WARN Bootstrap broker 192.168.0.28:9092 disconnected (org.apache.kafka.clients.NetworkClient)

If trying to physical host's port 9092.

Thanks for help and please ask for any further information needed to help solving this problem.

EDIT1:

Configuration changed to the following properties.

and kafka-broker is not running. server.log is showing following trace:

root@host1:/usr/hdp/2.6.3.0-235/kafka# cat /var/log/kafka/server.log
[2018-09-04 09:17:46,964] INFO KafkaConfig values:
        advertised.host.name = host1.ambari
        advertised.listeners = INTERNO://host1.ambari:6667,EXTERNO://192.168.0.28:9092
        advertised.port = 9092
        authorizer.class.name =
        auto.create.topics.enable = true
        auto.leader.rebalance.enable = true
        background.threads = 10
        broker.id = -1
        broker.id.generation.enable = true
        broker.rack = null
        compression.type = producer
        connections.max.idle.ms = 600000
        controlled.shutdown.enable = true
        controlled.shutdown.max.retries = 3
        controlled.shutdown.retry.backoff.ms = 5000
        controller.socket.timeout.ms = 30000
        default.replication.factor = 1
        delete.topic.enable = false
        fetch.purgatory.purge.interval.requests = 10000
        group.max.session.timeout.ms = 300000
        group.min.session.timeout.ms = 6000
        host.name =
        inter.broker.protocol.version = 0.10.1-IV2
        leader.imbalance.check.interval.seconds = 300
        leader.imbalance.per.broker.percentage = 10
        listeners = INTERNO://host1.ambari:6667,EXTERNO://host1.ambari:6667
        log.cleaner.backoff.ms = 15000
        log.cleaner.dedupe.buffer.size = 134217728
        log.cleaner.delete.retention.ms = 86400000
        log.cleaner.enable = true
        log.cleaner.io.buffer.load.factor = 0.9
        log.cleaner.io.buffer.size = 524288
        log.cleaner.io.max.bytes.per.second = 1.7976931348623157E308
        log.cleaner.min.cleanable.ratio = 0.5
        log.cleaner.min.compaction.lag.ms = 0
        log.cleaner.threads = 1
        log.cleanup.policy = [delete]
        log.dir = /tmp/kafka-logs
        log.dirs = /kafka-logs
        log.flush.interval.messages = 9223372036854775807
        log.flush.interval.ms = null
        log.flush.offset.checkpoint.interval.ms = 60000
        log.flush.scheduler.interval.ms = 9223372036854775807
        log.index.interval.bytes = 4096
        log.index.size.max.bytes = 10485760
        log.message.format.version = 0.10.1-IV2
        log.message.timestamp.difference.max.ms = 9223372036854775807
        log.message.timestamp.type = CreateTime
        log.preallocate = false
        log.retention.bytes = -1
        log.retention.check.interval.ms = 300000
        log.retention.hours = 168
        log.retention.minutes = null
        log.retention.ms = null
        log.roll.hours = 168
        log.roll.jitter.hours = 0
        log.roll.jitter.ms = null
        log.roll.ms = null
        log.segment.bytes = 1073741824
        log.segment.delete.delay.ms = 60000
        max.connections.per.ip = 2147483647
        max.connections.per.ip.overrides =
        message.max.bytes = 1000000
        metric.reporters = []
        metrics.num.samples = 2
        metrics.sample.window.ms = 30000
        min.insync.replicas = 1
        num.io.threads = 8
        num.network.threads = 3
        num.partitions = 1
        num.recovery.threads.per.data.dir = 1
        num.replica.fetchers = 1
        offset.metadata.max.bytes = 4096
        offsets.commit.required.acks = -1
        offsets.commit.timeout.ms = 5000
        offsets.load.buffer.size = 5242880
        offsets.retention.check.interval.ms = 600000
        offsets.retention.minutes = 86400000
        offsets.topic.compression.codec = 0
        offsets.topic.num.partitions = 50
        offsets.topic.replication.factor = 3
        offsets.topic.segment.bytes = 104857600
        port = 6667
        principal.builder.class = class org.apache.kafka.common.security.auth.DefaultPrincipalBuilder
        producer.purgatory.purge.interval.requests = 10000
        queued.max.requests = 500
        quota.consumer.default = 9223372036854775807
        quota.producer.default = 9223372036854775807
        quota.window.num = 11
        quota.window.size.seconds = 1
        replica.fetch.backoff.ms = 1000
        replica.fetch.max.bytes = 1048576
        replica.fetch.min.bytes = 1
        replica.fetch.response.max.bytes = 10485760
        replica.fetch.wait.max.ms = 500
        replica.high.watermark.checkpoint.interval.ms = 5000
        replica.lag.time.max.ms = 10000
        replica.socket.receive.buffer.bytes = 65536
        replica.socket.timeout.ms = 30000
        replication.quota.window.num = 11
        replication.quota.window.size.seconds = 1
        request.timeout.ms = 30000
        reserved.broker.max.id = 1000
        sasl.enabled.mechanisms = [GSSAPI]
        sasl.kerberos.kinit.cmd = /usr/bin/kinit
        sasl.kerberos.min.time.before.relogin = 60000
        sasl.kerberos.principal.to.local.rules = [DEFAULT]
        sasl.kerberos.service.name = null
        sasl.kerberos.ticket.renew.jitter = 0.05
        sasl.kerberos.ticket.renew.window.factor = 0.8
        sasl.mechanism.inter.broker.protocol = GSSAPI
        security.inter.broker.protocol = PLAINTEXT
        socket.receive.buffer.bytes = 102400
        socket.request.max.bytes = 104857600
        socket.send.buffer.bytes = 102400
        ssl.cipher.suites = null
        ssl.client.auth = none
        ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
        ssl.endpoint.identification.algorithm = null
        ssl.key.password = null
        ssl.keymanager.algorithm = SunX509
        ssl.keystore.location = null
        ssl.keystore.password = null
        ssl.keystore.type = JKS
        ssl.protocol = TLS
        ssl.provider = null
        ssl.secure.random.implementation = null
        ssl.trustmanager.algorithm = PKIX
        ssl.truststore.location = null
        ssl.truststore.password = null
        ssl.truststore.type = JKS
        unclean.leader.election.enable = true
        zookeeper.connect = host1.ambari:2181
        zookeeper.connection.timeout.ms = 25000
        zookeeper.session.timeout.ms = 30000
        zookeeper.set.acl = false
        zookeeper.sync.time.ms = 2000
 (kafka.server.KafkaConfig)
[2018-09-04 09:17:46,974] FATAL  (kafka.Kafka$)
java.lang.IllegalArgumentException: Error creating broker listeners from 'INTERNO://host1.ambari:6667,EXTERNO://host1.ambari:6667': No enum constant org.apache.kafka.common.protocol.SecurityProtocol.INTERNO
        at kafka.server.KafkaConfig.validateUniquePortAndProtocol(KafkaConfig.scala:994)
        at kafka.server.KafkaConfig.getListeners(KafkaConfig.scala:1013)
        at kafka.server.KafkaConfig.<init>(KafkaConfig.scala:966)
        at kafka.server.KafkaConfig$.fromProps(KafkaConfig.scala:779)
        at kafka.server.KafkaConfig$.fromProps(KafkaConfig.scala:776)
        at kafka.server.KafkaServerStartable$.fromProps(KafkaServerStartable.scala:28)
        at kafka.Kafka$.main(Kafka.scala:58)
        at kafka.Kafka.main(Kafka.scala)

Answer 1

If you're deploying Kafka within Docker, you need to configure the listener to be accessible if you want to be able to access it from outside your Docker network too. This article explains the details.