CODEWITHSUNDEEP
javaspringmultithreadingapache-kafkaspring-kafka
Rax
Rax

Reputation: 375

Achieving one consumer thread per kafka topic partition with spring kafka 2.5.8 release

I have been using apache kafka-clients(2.3.1 version to be precise) libraries for creating kafka consumers where the one partition - one consumer thread was achieved by following calculation:

no of consumer thread on a compute * no of computes = number of partitions for the topic

It used to be manual scale and hence when the number of computes needed to be crease, no of consumer threads running on one compute was accordingly decreased.

How do we achieve this using org.springframework.kafka.config.ConcurrentKafkaListenerContainerFactory.

I am trying to use spring kafka 2.5.8 release. The application is running on k8s cluster with auto scaling capability. Let's say if I have max and min pods set to 4, so ideally

4 X number of consumer threads = number of partitions for the topic

How is this number of consumer threads configured . Is it through this:

org.springframework.kafka.config.ConcurrentKafkaListenerContainerFactory.setConcurrency() . Can someone please guide.

Upvotes: 1

Views: 877

Answers (1)

Gary Russell
Gary Russell

Reputation: 174809

Yes; or the concurrency property on @KafkaListener which overrides the factory's concurrency.

If you change it at runtime, it won't take effect unless you stop() and start() the container.

Upvotes: 1

Related Questions