CODEWITHSUNDEEP
apache-kafkaspring-kafka
Bhuvan
Bhuvan

Reputation: 1

Spring Kafka, subscribing to large number of topics using topic pattern

Are there any known limitations with the number of Kafka topics and the topics distribution between consumer instances while subscribing to Kafka topics using topicPattern.

Our use case is that we need to subscribe to a large number of topics (potentially few thousands of topics). All the topics follow a naming convention and have only 1 partition. We don't know the list of topic names before hand and new topics that match the pattern can be created anytime. Our consumers should be able to consume messages from all the topics that match the pattern.

  1. Do we have any limit on the number of topics that can be subscribed this way using the topic pattern.

  2. Can we scale up the number of consumer instances to read from more topics

  3. Will the topics be distributed among all the running consumer instances. In our local testing it has been observed that all the topics are read by a single consumer despite having multiple consumer instances running in parallel. Only when the serving consumer instance is shutdown, the topics are picked by another instance (not distributed among available instances)

All the topics mentioned are single partition topics.

Upvotes: 0

Views: 701

Answers (1)

Gary Russell
Gary Russell

Reputation: 174554

  1. I don't believe there is a hard limit but...
  2. No; you can't change concurrency at runtime, but you can set a larger concurrency than needed and you will have idle consumers waiting for assignment.
  3. You would need to provide a custom partition assigner, or select one of the alternate provided ones e.g. RoundRobinAssignor will probably work for you https://kafka.apache.org/documentation/#consumerconfigs_partition.assignment.strategy

Upvotes: 1

Related Questions