CODEWITHSUNDEEP
dockerapache-kafkadocker-compose
Michael
Michael

Reputation: 2683

Use docker-compose with port mapping for local Kafka testing

I'm trying to get the infrastructure for a black box service test up and running using docker-compose. I want to do that on Jenkins, for multiple services, therefore it is necessary to bind to build-specific port (e.g. 50012 instead of 9092) for the build should be able to run in parallel.

The problem is, that producing messages fails.

Here is my docker-compose.yml:

---
version: "3.4"

services:

  zookeeper:
    image: "confluentinc/cp-zookeeper:latest"
    ports:
      - "2181"
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
    extra_hosts:
      - "moby:127.0.0.1"

  kafka:
    image: "confluentinc/cp-kafka:latest"
    environment:
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
      KAFKA_PROTOCOL_NAME: INSIDE
      KAFKA_ADVERTISED_PROTOCOL_NAME: OUTSIDE
      KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE
      KAFKA_LISTENERS: "OUTSIDE://localhost:50012,INSIDE://kafka:9092"
      KAFKA_ADVERTISED_LISTENERS: "OUTSIDE://localhost:50012,INSIDE://kafka:9092"
      KAFKA_BROKER_ID: 1
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_ZOOKEEPER_CONNECT: "zookeeper:2181"
    ports:
      - "50012:9092"
    extra_hosts:
      - "moby:127.0.0.1"
    depends_on:
      - zookeeper

I don't see any errors but this one (which I think can be ignored):

ERROR Could not submit metrics to Kafka topic __confluent.support.metrics: Failed to construct kafka producer (io.confluent.support.metrics.BaseMetricsReporter)

When I try to send a message with Kafka console producer like this

echo 'my-message' | kafka-console-producer.sh --broker-list localhost:50012 --topic test

I get this exception and the message does not get sent:

[2018-07-25 10:56:31,283] ERROR Error when sending message to topic test with key: null, value: 10 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)

org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms.

UPDATE: Thanks to the correct answer from Robin, we were able to get it running with following docker-compose.yml:

---
version: "3.4"

services:

zookeeper:
    image: "confluentinc/cp-zookeeper:latest"
    environment:
    ZOOKEEPER_CLIENT_PORT: 2181
    ZOOKEEPER_TICK_TIME: 2000
    extra_hosts:
      - "moby:127.0.0.1"

kafka:
    image: "confluentinc/cp-kafka:latest"
    environment:
    KAFKA_BROKER_ID: 1
    KAFKA_ZOOKEEPER_CONNECT: "zookeeper:2181"
    KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: "INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT"
    KAFKA_INTER_BROKER_LISTENER_NAME: "INSIDE"
    KAFKA_ADVERTISED_LISTENERS: "INSIDE://kafka:29092,OUTSIDE://localhost:50012"
    KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
    KAFKA_AUTO_CREATE_TOPICS_ENABLE: "true"
    ports:
      - "50012:50012"
    extra_hosts:
      - "moby:127.0.0.1"
    depends_on:
      - zookeeper

Upvotes: 5

Views: 5611

Answers (1)

Robin Moffatt
Robin Moffatt

Reputation: 32100

The problem with your config is 

      KAFKA_ADVERTISED_LISTENERS: "OUTSIDE://localhost:50012,INSIDE://kafka:9092"
[…]
    ports:
      - "50012:9092"

The KAFKA_ADVERTISED_LISTENERS is exactly that - the address that the broker advertises. So on one hand you're configuring it as localhost:50012, but then on the other you're exposing from your Docker config 9092 as the external port.

If you simply use

 ports:
      - "50012:50012"

then it should work fine.

Check out this docker-compose for an example of a working config that exposes Kafka externally and internally.

Upvotes: 7

Related Questions