CODEWITHSUNDEEP
dockershellapache-kafkadocker-compose
Nando Machado
Nando Machado

Reputation: 43

Create Kafka topic in Docker works from command line but not from Shell script file

I'm trying to create a topic in a Kafka container running in Docker using a shell script file.

docker-compose.yml

---
version: '2'
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:latest
    container_name: zookeeper
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
    ports:
      - "22181:2181"

  kafka:
    image: confluentinc/cp-kafka:latest
    container_name: kafka
    depends_on:
      - zookeeper

    ports:
      - "29092:29092"
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:29092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1

init.sh

#!/bin/zsh
docker compose up -d
docker exec kafka kafka-topics --create --bootstrap-server localhost:29092 --partitions 1 --replication-factor 1 --topic Test

When I run docker-compose the containers start-up fine, but the topic creation fails.

[2021-11-04 17:35:07,957] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (localhost/127.0.0.1:29092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)

However, when I run the same commands in the command line, without any changes, it works fine and creates the topic.

Created topic Test.

I'm at a loss and reading previous questions on similar issues here hasn't help: mostly they were due to typos or adding quotation marks where they are not needed. I have tried changing the #! to bash, sh, and usr/bin/env (amongst others) to no avail.

Any help appreciated.

Docker version: version 20.10.8 Running on a Mac (Intel)

Upvotes: 3

Views: 9376

Answers (4)

Nando Machado
Nando Machado

Reputation: 43

Everyone was right, I had to wait longer (I had tried 10 secs).

As @Luigi Cerone suggested, I used cub to wait for Kafka to be ready, by adding this line before creating a topic:

docker exec kafka cub kafka-ready -b localhost:29092 1 20

Upvotes: 1

Luigi Cerone
Luigi Cerone

Reputation: 372

As suggested by the error message, the broker is not available (in your case it hasn't started yet). You could wait for the broker to complete start up process (by looking at the logs with docker logs -f kafka or by using a sleep in your script) and then run your command to create the topic:

docker exec kafka kafka-topics --create --bootstrap-server localhost:29092 --partitions 1 --replication-factor 1 --topic Test

Alternatively, instead of having to figure out if your broker has completed the initialization process, you could use something like this official Confluent example. In this example, an utility is used to wait for a Kafka cluster to be ready.

Upvotes: 4

Jinna Baalu
Jinna Baalu

Reputation: 7839

Modify init.sh

Run the container, docker-compose up -d, wait for the broker to run. check with the logs if the application is up and running.

Now init.sh should have only create the topic command

#!/bin/zsh
docker exec kafka kafka-topics --create --bootstrap-server localhost:29092 --partitions 1 --replication-factor 1 --topic Test1

docker exec kafka kafka-topics --create --bootstrap-server localhost:29092 --partitions 1 --replication-factor 1 --topic Test2

docker exec kafka kafka-topics --create --bootstrap-server localhost:29092 --partitions 1 --replication-factor 1 --topic Test3

Upvotes: 0

OneCricketeer
OneCricketeer

Reputation: 191844

Where/when are you running the init.sh script? Maybe the broker hasn't started yet.

Also, use the internal listener when you want to run commands inside the Docker network

docker-compose exec kafka \
  bash -c "kafka-topics --create --bootstrap-server kafka:9092 --partitions 1 --replication-factor 1 --topic Test"

Upvotes: 0

Related Questions