CODEWITHSUNDEEP
pythonapache-kafkakafka-producer-api
Hamuel
Hamuel

Reputation: 633

Python KafkaProducer cannot set replication factor

As a fact when creating a topic in kafka it is possible to set the replication factor however I was using a KafkaProducer (the kafka api for python pip install kafka)

I thought I could do producer.send(...,replication-factor=3)but then there was no option for me to do that.

Now I have only one option left is to directly create a shell script to connect to kafka to create a topic but then if the feature of python kafka is so lackluster why would I continue using it. So is there a way to set a replication factor when I am going to produce a kafka topic.

Upvotes: 0

Views: 1303

Answers (1)

Mickael Maison
Mickael Maison

Reputation: 26875

kafka-python does not support the Kafka Admin APIs at the moment. The only way to create topics via this client is to rely on the auto-create broker feature.

However, as you've noticed, this does not allow you to provide any topic configurations.

You can either:

  • Set the replication factor in the broker config (that will apply to all topics) by setting default.replication.factor=3 in the broker's server.properties file.

  • Use a script (like the kafka-topics.sh tool) to explicitely create topics with custom settings.

The Kafka Admin APIs are still relatively new and very few clients apart from the official Java client support it.

Upvotes: 1

Related Questions