CODEWITHSUNDEEP
apache-kafkaapache-kafka-streams
Aditya Goel
Aditya Goel

Reputation: 241

Kafka Streams timeout rebalancing and out from cluster

Say We have 2 Stream-Tasks (running on different instances). The source topic have 1 partition. Upon start of application, the Partition-1 got allocated to Stream-Task-1. Now, Stream-1-task starts doing some processing and transformation and takes more than 10 minutes. Are there any possibilities that, Kafka framework would trigger rebalancing and just out the Stream-Task-1 from the system and allocates the P1 to Stream-Task-2 ??

Is there some property for stream-task, to control the time upto which rebalancing should NOT be trigger-red !!

Thanks for responses folks.

Upvotes: 0

Views: 289

Answers (1)

S Blee-G
S Blee-G

Reputation: 166

I assume by Stream-Task you just mean different instances of your Streams application? A "StreamTask" is an internal term that Streams uses to define a unit of work. If you have only one input topic with one partition, then you have one StreamTask in your application.

If one of your instances fails to poll within the poll interval then yes, it will get kicked out of the group and its task (ie partition) will be given to the other instance. If you think it might be stuck in processing for 10 minutes then you should adjust the max.poll.interval.ms config accordingly.

Upvotes: 2

Related Questions