CODEWITHSUNDEEP
apache-kafkakafka-consumer-apispring-kafka
dvlpr
dvlpr

Reputation: 371

Kafka Multiple consumers application

I have a kafka topic which has 50 partitions, and the publishing message rate is 500 a second. As port pf consumer design, we can either have Multiple consumers with their own threads or Single consumer, multiple worker processing threads using concurrent listeners.

Lets say each message takes 100ms to process. If we use a single consumer and use the concurrent Message Listener container then we will end up having 50 threads in a single deployment.

If we deploy multiple applications with same group id and assign each application 10 concurrent threads then we will require 5 deployments. Re balancing can be an issue if a one of the application restarts

Can you please suggest a good approach or splitting the topic is recommended if there are too many partitions

Upvotes: 2

Views: 2246

Answers (1)

Gary Russell
Gary Russell

Reputation: 174484

Yes, your understanding is correct 5 instances with concurrency 10 will have one consumer thread per partition. If you deploy only 2 instances, the partitions will be distributed across 20 threads.

You can mitigate rebalances by using the recent static group membership feature.

group.instance.id

A unique identifier of the consumer instance provided by the end user. Only non-empty strings are permitted. If set, the consumer is treated as a static member, which means that only one instance with this ID is allowed in the consumer group at any time. This can be used in combination with a larger session timeout to avoid group rebalances caused by transient unavailability (e.g. process restarts). If not set, the consumer will join the group as a dynamic member, which is the traditional behavior.

Upvotes: 1

Related Questions