CODEWITHSUNDEEP
apache-kafkakafka-producer-apiapache-kafka-streams
user3735823
user3735823

Reputation: 41

Kafka Streams state store backup topic partitioning strategy

Kafka guarantees that messages with same key will always go to the same partition. For instance, I have message with the string key: 2329. And two topics t1 and t2. When I perform write of this message it goes into partition 1 in both topics, as expected.

Now the problem itself: I'm using Kafka Streams 0.10.2.0 persistent state store, which automatically creates a backup topic. Now in case of this backup topic message with the key: 2329 goes into another partition (partition 0), which is strange to me.

Has anyone encountered this issue?

Upvotes: 1

Views: 607

Answers (1)

user3735823
user3735823

Reputation: 41

I've found where was the issue.

To increase performance I've skipped repartitioning before write data to the state store. And used another column from the value as the state store key. And it worked until additional topic with enrichment information has been added. So I just forgot to perform repartitioning.

Upvotes: 3

Related Questions