CODEWITHSUNDEEP
apache-kafkaapache-kafka-streams
Chris
Chris

Reputation: 1349

Kafka Streams Internal Topic Naming

According to the documentation (https://docs.confluent.io/current/streams/developer-guide/manage-topics.html#internal-topics), internal topics follow the naming convention <application.id>-<operatorName>-<suffix>.

Some examples we have are:

testapplication-KSTREAM-REDUCE-STATE-STORE-0000000008-repartition  
testapplication-KSTREAM-REDUCE-STATE-STORE-0000000027-repartition

Does anyone know how the integer are determined?

Unfortunately our security requirements do not allow for us to create topics with our applications and need to be setup ahead of time. I am trying to determine if these topic names will be consistent.

Upvotes: 7

Views: 8808

Answers (5)

sparrovv
sparrovv

Reputation: 7784

To answer your main question about the integer, that's what I found in docs:

The number is a globally incrementing number that represents the operator’s order in the topology. The generated number is prefixed with a varying number of “0”s to create a string that is consistently 10 characters long.

This is quite important aspect of kafka streams DSL and can lead to some problems if you change a topology. It's a good practice to name your stateful operator.

More information you can find in dsl-topology-naming article

Upvotes: 1

Dominic Douglas
Dominic Douglas

Reputation: 1

To answer your question about topic names being consistent, in my experience, they have been consistent between executions of the application, however, if you modify the order of, add, or delete any joins or reduces within your logic, the topic names may change.

Upvotes: 0

Vassilis
Vassilis

Reputation: 1054

Have you seen these commands to set ACL for Streams internal topics. I believe they are introduced as part of Kafka v2.x.x (confluent doc)

   # Allow Streams to manage its own internal topics and consumer groups:
   bin/kafka-acls ... --add --allow-principal User:team1 --operation All --resource- 
   pattern-type prefixed --topic team1-streams-app1 --group team1-streams-app1

So you just need to know the steams application.id, which is the prefix of all internal topics.

I believe as you will give permission ALL, that would allow their creation as well.

Upvotes: 0

Milos Pajic
Milos Pajic

Reputation: 336

Integers are internally generated.

You can find it documented here:

https://docs.confluent.io/current/streams/javadocs/index.html

Under groupBy method descriptions it says:

Because a new key is selected, an internal repartitioning topic will be created in Kafka. This topic will be named "${applicationId}-XXX-repartition", where "applicationId" is user-specified in StreamsConfig via parameter APPLICATION_ID_CONFIG, "XXX" is an internally generated name, and "-repartition" is a fixed suffix. You can retrieve all generated internal topic names via Topology.describe().

Upvotes: -1

Nishu Tayal
Nishu Tayal

Reputation: 20840

Usually the intermediate topic names are constructed with following convention:

<ApplicationId>-<operator name>-<suffix>

Suffix value can be either "changelog" or "repartition"

Based on the operator, it uses one of the suffix. Here is an example:

testapplication-aggregate-repartition

testapplication-aggregate-changelog

Upvotes: 3

Related Questions