CODEWITHSUNDEEP
spring-cloud-stream
Ajay
Ajay

Reputation: 485

Spring Cloud Stream Kafka consumer patterns

For a topic with multiple partitions -

1) Does a single SpringBoot instance use multiple threads to process (method annotated with StreamListener) each message from each partition?

2) Is it possible to configure more than one thread for each partition or is that something I would have to manually hand off from my listener thread to a worker pool?

Upvotes: 7

Views: 2755

Answers (1)

Gary Russell
Gary Russell

Reputation: 174554

....consumer.concurrency controls the number of threads (default 1).

The partitions are distributed across the threads. If you have 20 partitions and 4 threads; they'll get 5 partitions each.

You need to have at least as many partitions as the aggregate concurrency across all instances. (If you have 2 app instances and 5 threads each, you need at least 10 partitions).

You should not distribute messages from a single partition across multiple threads; the offset will be committed as soon as you hand off to the new thread and that could cause message loss.

You should always err on the side of having more partitions than you need concurrency.

Upvotes: 7

Related Questions