CODEWITHSUNDEEP
apache-kafka
Gourav
Gourav

Reputation: 55

Broadcasting message to all subscriber within same consumer group

Using apache-Kafka version 2.11-2.4.0.

I am creating one app where I want to publish the event to all my subscribers (consumers) in the same consumer group. But in many places, I found that Kafka broadcast messages to the consumer group. Is there is any way to achieve this?

Upvotes: 1

Views: 1813

Answers (2)

Jon Bates
Jon Bates

Reputation: 3173

You can achieve this by sending a copy of your message to every partition manually. You can use the admin client, and do something like this (assuming java) to get all partitions for a topic:

List<String> topicName = Arrays.asList("topic-name");
List<TopicPartitionInfo> partitions = 
        adminClient.describeTopics(topicName)
                   .values()
                   .get("topic-name")
                   .get()
                   .partitions();

The partitions list contains an object for each partition, which can be used to explicitly produce a record to

partitions.stream().forEach(x -> {
    ProducerRecord<Key, Value> record = new ProducerRecord<>("topic-name", x, someKey, someValue);
    kafkaProducer.send(record);
});

As ever, this isn't production-grade code, just an illustration. Also be aware that if you have a consumer subscribed to multiple partitions, they will receive your message multiple times

Upvotes: 0

H.&#199;.T
H.&#199;.T

Reputation: 3559

In Kafka consumer group is just for horizontal scalability. For example if you have 5 consumers in same consumer group and subscribe a topic that has 5 partitions then Kafka assign each partition to a consumer in the group. So messages are consumed in parallel by consumers.

In your case if you want to consume same messages from multiple consumers, then you need to have multiple consumer groups that subcsribe the same topic. You can achive this by setting different group.id for your consumers.

Note: As a good practice your number of consumers in a consumer group should be the same as the number of partitions of the topic which is subscribed.

Upvotes: 2

Related Questions