CODEWITHSUNDEEP
apache-kafkakafka-consumer-apiapache-kafka-streams
Michał Szewczyk
Michał Szewczyk

Reputation: 8158

Kafka Streams - disable internal topic creation

I work in an organization where we must use the shared Kafka cluster.
Due to internal company policy, the account we use for authentication has only the read/write permissions assigned.
We are not able to request the topic-create permission.
To create the topic we need to follow the onboarding procedure and know the topic name upfront.

As we know, Kafka Streams creates internal topics to persist the stream's state.

Is there a way to disable the fault tolerance and keep the stream state in memory or persist in the file system?

Thank you in advance.

Upvotes: 0

Views: 721

Answers (2)

OneCricketeer
OneCricketeer

Reputation: 191983

This entirely depends how you write the topology. For example, map/filter/forEach, etc stateless DSL operators don't create any internal topics.

If you actually need to do aggregation, and build state-stores, then you really shouldn't disable topics, assuming you could. Yes, statestores are stored either in-memory or as RocksDB on disk, but they're still initially stored as topics so they can actually be distributed, or rebuilt in case of failure.

If you want to prevent them, I think you'll need an authorizer class defined on the broker that can restrict topic creation based, at least, on client side application.id and client.id regex patterns, but there's nothing you can do at the client config.

Upvotes: 2

Matthias J. Sax
Matthias J. Sax

Reputation: 62350

Kafka Streams cannot work without these internal topics, and thus, you cannot disable topic creation. However, you can create them upfront, and thus, the application won't need create permissions (if the topics exists, the app won't send a create topic request).

You can use Topology#describe() to find out about internal topic names, and request that those topics are create beforehand. Note: the topics will need to correct number of partitions (as a rule of thumbs, the same number of partitions as your input topic). If the partitions count is not as expected, the app won't start up with an error message (which should contain the expected number of partitions, so you could fix it if you did not get it right the first time).

Upvotes: 0

Related Questions