CODEWITHSUNDEEP
javaapache-kafkakafka-consumer-api
Joker
Joker

Reputation: 11156

Is there a limitation on the number of consumer to the same topic in Kafka?

Does Kafka have a limitation on the simultaneous connections (created with Consumer.createJavaConsumerConnector) for the same topic within the same group?

My scenario is I need to consume a topic from different process (not thread), so I need to create lots of high level consumers.

Upvotes: 0

Views: 433

Answers (2)

alirabiee
alirabiee

Reputation: 1306

The number of active consumers within the same consumer group is limited by the number of partitions of the topic. Extra consumers will act as backups and will only start consuming when one of the active consumers goes down and the group is re-balanced.

If you need to consume the same copy of the data within multiple processes, your consumers should be in different consumer groups. There is no limitation on the number of consumer groups you can have.

Upvotes: 2

Alex Ott
Alex Ott

Reputation: 87164

The main limitation is number of partitions that have this topic - you can create more consumers than partitions, but they won't consume anything.

Upvotes: 0

Related Questions