CODEWITHSUNDEEP
javaapache-kafkaapache-kafka-streams
ethrbunny
ethrbunny

Reputation: 10479

Kafka KStreams - how to add threads / using StreamsConfig.NUM_STREAM_THREADS_CONFIG

I was messing about with this param and ran into some oddness. My app runs ok without it but when I added this line to the config:

 config.put(StreamsConfig.NUM_STREAM_THREADS_CONFIG, "3");

CPU usage wouldn't climb above zero. App didn't appear to be doing anything. No errors.

Is there some advised method to increasing thread usage for a KStreams app? Or just 'trust the force' and let it all run together?

Edit:

  1. I have two partitions
  2. Have checked the consumer lag via kafka-consumer-groups - lots of records available
  3. Even if there were only 1 partition - why would having multiple threads do nothing? 0% CPU.

Upvotes: 2

Views: 4814

Answers (1)

Damian Guy
Damian Guy

Reputation: 131

How many partitions do you have? If you only have a single partition then increasing the number of threads wont have any effect as the number of partitions defines the maximum parallelism. So if you have 1 partition and 3 threads, you'll only have 1 busy thread.

Check there is data available for consumption on the input topics. Make sure you have StreamsConfig.AUTO_OFFSET_RESET_CONFIG set to latest. If you ran it previously with the same applicationId then Kafka Streams may have already consumed all of the data, so there will be nothing to do. In this case you could use a different applicationId or you could use the Kafka Streams Reset Tool to reset the topics.

This setting is used in some of the tests, i.e.,KStreamRepartitionJoinTest and seems to be working fine.

Upvotes: 7

Related Questions