CODEWITHSUNDEEP
additionconsumerkafka-consumer-apiapache-kafka
Vladimir Nabokov
Vladimir Nabokov

Reputation: 1907

Kafka consumer - how to add a topic

In a scenario, where I have N consumers (all consumers have 1 stream/no partitioning), each subscribed to separate set of topics, how to process new topics, added by producers?

Should I create new consumer to each added topic? Or can I add this topic to already working consumer? (how to do that?)

Or better keep 1 consumer group with N consumers and not divide topics between N consumers(streams)?

Upvotes: 0

Views: 1857

Answers (1)

Vladimir Nabokov
Vladimir Nabokov

Reputation: 1907

The initial config, as described, where topics devided between consumers of the same consumer group is not goood at all. Consumer will not be able to fail=over each other, because they have different lists of topics. Kafka will issue unnecessary rebalance.

You need not add consumer for each added topic - you better consume all such topics from beginning, adding them to common path like /MYTOPICS/*

You better divide topics by partitions and add new partitions if you need. This will be transparent and without effort.

Upvotes: 1

Related Questions