CODEWITHSUNDEEP
apache-kafkaapache-kafka-streams
Raman
Raman

Reputation: 717

Kafka Stream: What happens to processor object when REBALANCE occur

1) We use Low level processor API. When the rebalance occurs does the processor object gets re-initialized and init method invoked again?

2) Is REBALANCE partition specific (single application having multiple threads consuming different partitions) OR for the whole Kafka stream, I assume later since it is a Stream state?

Upvotes: 1

Views: 157

Answers (1)

Anand Sai
Anand Sai

Reputation: 1586

1) We use Low level processor API. When the rebalance occurs does the processor object gets re-initialized and init method invoked again?

In older versions (2.3.x or older), basically, when a StreamThread is started, it first triggers a rebalance, and after partitions are assigned, tasks are created and corresponding init() calls are made. For existing StreamThreads when a rebalance is triggered, all tasks are suspended (ie, calls to close() are made) and re-assigned as well as new tasks are started again.

In newer versions (2.4.x and newer), an incremental rebalancing is done and thus tasks are not suspended during a rebalance any longer.

2) Is REBALANCE partition specific (single application having multiple threads consuming different partitions) OR for the whole Kafka stream, I assume later since it is a Stream state?

You are right!! As quoted in this blog:

Kafka Streams uses it to assign tasks and partitions to the application streams instances.

This blog talks about Kafka rebalancing and it's usage to assign to application stream instances.

Let me know if it helps!!

Upvotes: 2

Related Questions