CODEWITHSUNDEEP
apache-kafka-streams
faith
faith

Reputation: 13

Stream thread calculation

I'm using Stream DSL. I have three source topic with partition 17, 100, 40. I will be running three instances and 2 standby instances.

How can I calculate how many stream threads I will need so that each thread gets exactly one task or highest parallelism is achieved?

Upvotes: 0

Views: 88

Answers (2)

Sriram
Sriram

Reputation: 1

I will try to discuss an approach here in short

You are asking for maximum parallelism

  • This can be achieved by separating out each topic in a separate topology
  • Each topology having separate thread count (one thread per consumer per topic) - 17/3, 100/3, 40/3 - topic partition/instances
  • This will make sure that each topology gets separate thread count and separate parallelism
  • each topology will act as separate consumer group

Upvotes: 0

Matthias J. Sax
Matthias J. Sax

Reputation: 62350

This depends on the structure of your application. You can run the application with a single thread and observe the number of created tasks. The number of task is the maximum number of threads you can use.

The task that are created are logged or you obtain them via KafkaStream#localThreadMetadata().

Upvotes: 1

Related Questions