CODEWITHSUNDEEP
apache-kafkaapache-kafka-streams
rishi007bansod
rishi007bansod

Reputation: 1469

Setting Different Threads for Different Operations within Kafka Streams

I have defined following topology in kafka streams 

Operation 1 : input_stream ----> filter ----> window_processing ----> write_to_topic
Operation 2 : input_stream ----> write_to_topic

I have observed that both operations are being performed by same thread(even if I increase StreamsConfig.NUM_STREAM_THREADS_CONFIG factor for threading). But now I want them to be processed by different threads as Operation 1 is slowing down Operation 2. Is there way way I can set different threads for different Operations. As in above case Operation 1 is blocking and slowing down Operation 2.

Upvotes: 2

Views: 491

Answers (1)

Matthias J. Sax
Matthias J. Sax

Reputation: 62285

Kafka Streams parallelizes via partitions and sub-topologies/tasks. In your case, there is only one sub-topology and thus you can only parallelize via partitions. Thus, you can run as many threads as you have input topic partitions in parallel and each thread processed one partition (or multiple partitions is you have less threads than partitions).

If you really want to split both "operation branches" into independent thread, you need to write two applications instead of one.

Upvotes: 1

Related Questions