CODEWITHSUNDEEP
apache-kafka
user3018350
user3018350

Reputation:

Kafka: Routing of messages within topic

I am integrating Kafka into our system where we have one central component which is publishing same type of event to specific topic. On consumer side we have multiple 'projects' which are interested in events from that topic that are only relevant to them. Is there a way to deliver only relevant events to those consumers other then following

  1. Creating consumer group for each project and just ignoring events from other projects on application level.
  2. Creating separate topic for each project

This might be rookie question :/ Thanks in advance

Upvotes: 1

Views: 2683

Answers (2)

Aritra Gupta
Aritra Gupta

Reputation: 9

Based on your use case, I think partitioning based on your 'projects' on the consumer side might help in this case. The central producer can push to the same topic in the relevant partitions for the consumers to consume from.

At the consumers level, you need to keep all the consumers under one consumer group and specify which partition is to be consumed by respective consumers. This approach will have one negative impact - load balancing won't be taken care automatically by Kafka as we manually map each consumer to each partition.

Upvotes: 0

Mickael Maison
Mickael Maison

Reputation: 26865

The 2 options you listed can work but that makes you change the way your input and output clients work.

Another alternative is to use Kafka Stream to read all messages from the input topic and send each of them to the desired output topics. That way the producing side stays the same and your consumers only receive messages they care about.

Take a look at the KStream.branch() transformation.

Upvotes: 1

Related Questions