CODEWITHSUNDEEP
apache-kafkakafka-consumer-apispring-kafka
Neer1009
Neer1009

Reputation: 314

How to decide the concurrency to be set in spring kafka?

I am writing a kafka consumer using @KafkaListener annotation and i got to know that there is a way we can increase the number of concurrent kafka consumers from different partition using a method in ConcurrentKafkaListenerContainerFactory

e.g. factory.setConcurrency(3);

Javadoc for setconcurrency says like this:-

The maximum number of concurrent KafkaMessageListenerContainer running. Messages from within the same partition will be processed sequentially.

Now my question is

I have a kafka topic with 144 partitions to which our application needs to consume the message and 3 instance of app is running in parallel.

I want to know how to decide the concurrency value needs to bet set in 

ConcurrentKafkaListenerContainerFactory.setconcurrency (<Value>)

so that we can achieve high throughput in consuming the message.

should i use 144/3 = 48 as concurrency factor or is there a formula to derive this number ?

Upvotes: 7

Views: 13856

Answers (1)

Ryuzaki L
Ryuzaki L

Reputation: 39978

Yes the best you have is setting concurrency to 48 in each instance so that each partition will be consumed from unique thread in consumer group, And also to achieve high throughput you can use Batch listeners with higher batch size

The another best option is having more instance running for example 14 and each having concurrency level of 10. In both the approaches you also need to consider the available CPU for each instance having over head threads than CPU will not give better performance

Starting with version 1.1, you can configure @KafkaListener methods to receive the entire batch of consumer records received from the consumer poll. To configure the listener container factory to create batch listeners, you can set the batchListener property

Upvotes: 3

Related Questions