I need to create a kafka image with topics already created

Question

I have a requirement that i need to setup kafka locally with topics already there in the container.I am using ladoop/fast-data-dev for doing that

How manually i am doing it-

docker run -d --name landoopkafka -p 2181:2181 -p 3030:3030 -p 8081:8081 -p 8082:8082 -p 8083:8083 -p 9092:9092 -e ADV_HOST=localhost landoop/fast-data-dev

After running this command my container is up and running.

now i go to bash inside this container like docker -exec -it landopkafka bash

and create topic using this command

kafka-topics --zookeeper localhost:2181 --create --topic hello_topic --partitions 1 --replication-factor 1

My topic is created.

But my requirement is i need to have a docker file which will have topic created and i just need to run it.

OR

A docker compose file which i need to run

Guys i need help on this ,as i am absolutely new to docker and kafka

Answer 1

I had to do it too! What if I did not want to use wurstmeister images? I decided to make a custom script which will do the job, and run this script in a separate container.

Repository

https://github.com/yan-khonski-it/kafka-compose

Note, it will work with kafka versions that use zookeeper. Is Zookeeper a must for Kafka?

To start kafka with all your topics and zookeeper - docker-compose up -d.

---

Implementation details.

docker-compose.yml

# These services are kafka related. This docker-compose allows to start kafka locally quickly.

version: '2.1'

networks:
  demo-network:
    name: demo-network
    driver: bridge

services:
  zookeeper:
    image: "confluentinc/cp-zookeeper:${CONFLUENT_PLATFORM_VERSION}"
    container_name: zookeeper
    environment:
      ZOOKEEPER_CLIENT_PORT: 32181
      ZOOKEEPER_TICK_TIME: 2000
    ports:
      - 32181:32181
    hostname: zookeeper
    networks:
      - demo-network

  kafka:
    image: "confluentinc/cp-kafka:${CONFLUENT_PLATFORM_VERSION}"
    container_name: kafka
    hostname: kafka
    ports:
      - 9092:9092
      - 29092:29092

    environment:
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:32181
      KAFKA_BROKER_ID: 1
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092,PLAINTEXT_HOST://kafka:29092
      LISTENERS: PLAINTEXT://0.0.0.0:9092
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1

    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    depends_on:
      - "zookeeper"
    networks:
      - demo-network

  # Automatically creates required kafka topics if they were not created.
  kafka-topics-creator:
    build:
      context: kafka-topic-creator
      dockerfile: Dockerfile
    container_name: kafka-topics-creator
    depends_on:
      - zookeeper
      - kafka
    environment:
      ZOOKEEPER_HOSTS: "zookeeper:32181"
      KAFKA_TOPICS: "topic_v1 topic_v2"
    networks:
      - demo-network

Then I have a directory kafka-topics-creator. Here, I have three files create-kafka-topics.sh, Dockerfile, README.md.

Dockerfile

# It is recommened to use same version as kafka broker is used.
# So no additional images are pulled.
FROM confluentinc/cp-kafka:4.1.2

WORKDIR usr/bin

# Once it is executed, this container is not needed.
COPY create-kafka-topics.sh create-kafka-topics.sh
ENTRYPOINT ["./create-kafka-topics.sh"]

create-kafka-topics.sh

#!/bin/bash

# Simply wait until original kafka container and zookeeper are started.
sleep 15.0s

# Parse string of kafka topics into an array
# https://stackoverflow.com/a/10586169/4587961
kafkatopicsArrayString="$KAFKA_TOPICS"
IFS=' ' read -r -a kafkaTopicsArray <<< "$kafkatopicsArrayString"

# A separate variable for zookeeper hosts.
zookeeperHostsValue=$ZOOKEEPER_HOSTS

# Create kafka topic for each topic item from split array of topics.
for newTopic in "${kafkaTopicsArray[@]}"; do
  # https://kafka.apache.org/quickstart
  kafka-topics --create --topic "$newTopic" --partitions 1 --replication-factor 1 --if-not-exists --zookeeper "$zookeeperHostsValue"
done

README.md - so other people know how to use it.Always document your stuff - good advise.

# Creates kafka topics automatically.

## Parameters
`ZOOKEEPER_HOSTS` - zookeeper hosts,  I used value `"zookeeper:32181"` to run it locally.

`KAFKA_TOPICS` - space separated list of kafka topics. Example, `topic_1, topic_2, topic_3`.

Note, this container should run only **after** your original kafka broker and zookeeper are running.
After this container creates topics, it is not needed anymore.

How to check that the topics were created.

One solution is to check logs of kafka-topics-creator container.

docker logs kafka-topics-creator should print

$ docker logs kafka-topics-creator
WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both.
Created topic "topic_v1".
WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both.
Created topic "topic_v2".

Answer 2

You can create a docker-compose file like this...

version: '2'

services:
  zookeeper:
      image: wurstmeister/zookeeper:latest
      ports:
        - "2181:2181"

  kafka:
      image: wurstmeister/kafka:0.10.2.1
      ports:
        - "9092:9092"
      environment:
        KAFKA_ADVERTISED_HOST_NAME: 127.0.0.1
        KAFKA_CREATE_TOPICS: "MY_TOPIC_ONE:1:1,/
                              MY_TOPIC_TWO:1:1"
        KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      volumes:
        - /var/run/docker.sock:/var/run/docker.sock

Put your topics there and run docker-compose up

Answer 3

You should instead try to use the wurstmeister/kafka image which supports an environment variable to create topics during container startup.

Sure, the Landoop container has a bunch of other useful things, but sounds like you only want Kafka and don't want to mess with editing any Dockerfiles

The other solution is to startup a second container after Kafka which runs the create scripts, then stops itself