CODEWITHSUNDEEP

apache-kafka-streams

RyanShao

Reputation: 425

what is the maximum tasks for kafka streams on multiple topcis with different partition

Suppose we have two input topics. Topic1 has 2 partitions, and Topic2 has 4 partitions.
We create the kafka stream application with thread number 1.

Question: what is the maximum number that we can run the stream application that all will be assigned a partition?

as for my understanding, it is decided by the maximum partition of input topics. that is 4.

while what I want to achieve is 6, that is the sum number of all topics' partition. do you know is this doable? Thanks.

Upvotes: 0

Views: 300

Answers (1)

Reputation: 730

The parallelism of a streams application is defined number of partitions in the input topic(s), you are correct. You cannot change this. A workaround would be to work with an intermediate repartition topic: you repartition the input topic into a new topic with 6 partitions, and then do the actual work with a parallelism of 6.

Upvotes: 1

Related Questions