CODEWITHSUNDEEP
apache-kafkaapache-kafka-streams
pl0u
pl0u

Reputation: 375

KafkaStream createTopic not respecting Kafka server's auto.create.topics.enable settings

We have a production Kafka cluster which recently got polluted with a bunch of new topics. The Kafka cluster has the following settings:

auto.create.topics.enable=false
delete.topic.enable=false

Upon investigation, I found that these topics were created by a client team who is using KafkaStream's createTopic method: org.apache.kafka.streams.integration.utils.EmbeddedKafkaCluster#createTopic(java.lang.String, int, int)

Does this imply that KafkaStream's topic creation doesn't go through the Server side broker setting of auto.create.topics.enable? Does this mean that the createTopic* methods do not count as auto-topic creation? If so, how can we stop client teams from programmatically creating topics on a Kafka cluster?

edit: the kafka cluster is running 10.1.1 and the client is running 1.0.0 for both Kafka and Kafka-Stream

Upvotes: 5

Views: 5793

Answers (1)

Matthias J. Sax
Matthias J. Sax

Reputation: 62285

The config auto.create.topics.enable only applies if a client tries to read/write from/to a non-existing topic (or queries for metadata for a non-existing topic).

Kafka Streams, does create topic explicitly with a CreateTopic request and thus auto.create.topics.enable does not apply.

You can set up some ACLs for your cluster to control who can create topics: https://kafka.apache.org/0101/documentation.html#security_authz

Note: if you disallow the team that uses Kafka Streams to create new topics, you make it very hard for them to use Kafka Streams. Kafka Streams cannot operate without those topics and you are required to create those topics manually (with correct configurations) to not break the Kafka Streams application.

Upvotes: 7

Related Questions