CODEWITHSUNDEEP
apache-kafkakafka-consumer-api
xabhi
xabhi

Reputation: 878

What is the recommended number of consumers per group in Kafka?

What is the recommended number of consumers per group in Kafka? Basically, I have 400 topics, i want to consume from in my group. I noticed in my experimentation that I if I 1 consumer to consumer all 400 topics, it works. But if I create 1 consumer for each topic in a single consumer group, I am not able to receive any messages.

EDIT: I found that some consumers are able to receive messages during the poll loop but not all them. My guess is that they are being removed from the consumer group due to missing heartbeat.

From the logs it looks like after few minutes some consumer are not able to send any heartbeats because the group is always rebalancing:

[2019-03-08 07:42:36,769] INFO [Consumer clientId=jira_con-chow-test-69, groupId=chow] Attempt to heartbeat failed since group is rebalancing (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2019-03-08 07:42:39,671] DEBUG [Consumer clientId=jira_con-chow-test-69, groupId=chow] Sending Heartbeat request to coordinator mwkafka-prod-02.tbd:9092 (id: 2147483641 rack: null) (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
....

I want to know practically what is the appropriate number of consumers that should join a group. Or Am I interpreting this problem incorrectly and I should look at something else.

Thanks, Abhi

Upvotes: 2

Views: 5859

Answers (1)

Ashish Bhosle
Ashish Bhosle

Reputation: 657

The ideal equation is: number of partitions = number of consumers in a consumer group.

Consider, a topic t with 50 partitions has a consumer group of 10 consumers, then it will (at first) start consuming 10 partitions only. After reaching last offset in a partition, the consumers start consuming other partitions of the topic t.

For more details Kafka Consumer

Another useful post

Thanks

Upvotes: 0

Related Questions