CODEWITHSUNDEEP
dockergroovyapache-kafka
WILLIAM WOODMAN
WILLIAM WOODMAN

Reputation: 1231

how do i add a topic to a running kafka container using docker commands?

I have a kafka container started using following 

docker run --detach --name kafka -p 2181:2181 -p 9092:9092 --env ADVERTISED_HOST=192.168.1.89 --env ADVERVTISED_PORT=9092 --env AUTO.CREATE.TOPICS.ENABLE spotify/kafka

i can use docker logs kafka to see its started.

I then created a simple groovy script client producer to write some entries, however this keeps erroring with 

> Sending metadata request {topics=[wills topic]} to node 0
> Error while fetching metadata with correlation id 1 : {wills topic=INVALID_TOPIC_EXCEPTION}
....

I have set the following properties in the client code 

    Properties props = new Properties()
    props.put("bootstrap.servers", "192.168.1.89:9092" )   //Assign localhost id and external port (9092 int)
    props.put("acks", "all")                            //Set acknowledgements for producer requests.
    props.put("retries", 0)                             //If the request fails, the producer can automatically retry,
    props.put("batch.size", 16384)                      //Specify buffer size in config
    props.put("linger.ms", 1)                           //Reduce the no of requests less than 0
    props.put("buffer.memory", 33554432)                //The buffer.memory controls the total amount of memory available to the producer for buffering.
    props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer")
    props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer")
    props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer")
    props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer")
    props.put ("auto.create.topics.enable", true)       //enable auto topic creation

    producer = new org.apache.kafka.clients.producer.KafkaProducer<String, String>(props)

    for(int i = 0; i < 10; i++) {
        producer.send(new ProducerRecord<String, String>(topicName, Integer.toString(i), Integer.toString(i)))
    }
    println("Message sent successfully")
    producer.close()

but that does not not seem to enable auto topic creation. Given that i've spun the kafka server up as docker container - how does one administer that container instance once it is up and running using docker commands as the means to talk to the container

there is a note on the kafka site along the line of running a shell commend "bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test"

but its not easy to see how one edits/interacts with this in container once its up.

how does one administer the kafka container instance in docker once it has been started. otherwise what to you have to add to the docker run command to get topics pre created at startup

Advice greatfully received at this point

Upvotes: 9

Views: 27185

Answers (4)

Loathian
Loathian

Reputation: 1247

I thought I'd post and mention another way if you are using a docker container, specifically the wurstmeister Kafka image. The solution above that uses a shell script didn't work for me as Kafka was not running in the container yet when the command was executed.

Instead, I used these environment variables in the docker-compose.yml file

KAFKA_PORT: 30000 # necessary for KAFKA_CREATE_TOPICS to work
KAFKA_CREATE_TOPICS: "topic1:1:1"

Upvotes: 0

Amit Jain
Amit Jain

Reputation: 127

I have tried the above steps but facing some permission problems there is one more solution you can try in mine it worked perfectly

  1. Download Apache Kafka binary file from the official website https://kafka.apache.org/downloads.

  2. Open the terminal go through the path up to the folder. For eg /Users//Downloads/kafka_2.13-3.0.0

  3. And run this command

    bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic 'topic-name'

If you want to create multiple topics with a single statement only

  1. Create a text file first and save it topics.txt in Kafka folder in a format like ::

Eg:-

topic1:1:1

topic2:1:1

  1. Open terminal up to the downloaded Kafka folder

  2. Run the command with the help of awk

    awk -F':' '{ system("./bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --topic=" $1 " --partitions=" $2 " --replication-factor=" $3) }' /topics.txt

Upvotes: 1

Ayeshmantha Perera
Ayeshmantha Perera

Reputation: 909

use docker exec -it [container_id] /opt/kafka_2.11-0.10.1.0/bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test

Upvotes: 5

Sebastian
Sebastian

Reputation: 17443

The error message {wills topic=INVALID_TOPIC_EXCEPTION} is indicating that you are using wills topic as topic name.

A topic name cannot contain whitespaces. Try to rename to wills_topic and it should solve the problem.

This regex describes the legal characters for topic names (check sources):

val legalChars = "[a-zA-Z0-9\\._\\-]"

Use docker exec -it <container-name> <command> to launch the Kafka admin tools. Or just open a bash within your container docker exec -it <container-name> bash (see Docker docs).

Upvotes: 3

Related Questions