CODEWITHSUNDEEP
apache-kafkakafka-consumer-apiapache-kafka-streamsspring-cloud-stream
Bhavesh Shah
Bhavesh Shah

Reputation: 3389

How to make sure multiple Kafka topologies have same partition assignment of all topics?

I have an use case where we have multiple Kafka topologies defined in same application. I am running multiple instances of my application (e.g. 2 topologies running on 2 instances). Let's say 1st topology have 2 topics (t0, t1) with 2 partitions (p0, p1) to consume and 2nd topology have 2 topics (t2, t3) with 2 partitions (p0, p1) to consume and all the topics will have same key values (e.g. eventId). I could see that the partitions of both the topics of 1st & 2nd topology are getting assigned to a respective consumer.

Like for 1st toplogy t0-p0, t1-p0 is assigned to consumer-0 and t0-p1, t1-p1 is assigned to consumer-1. Like for 2nd toplogy t2-p0, t3-p0 is assigned to consumer-1 and t2-p1, t3-p1 is assigned to consumer-0.

Now the thing which I wanted to make sure is that parition-0 of all the topics in their respective topology should always gets assigned to consumer-0. But I could see that in my use case, in 1st topology, t0-p0, t1-p0 are assigned to consumer-0, whereas in 2nd topology, t2-p0, t3-p0 are assigned to consumer-1.

Is there any way by which I can configure my application to always assign the consumer-0 for the 0th partition and consumer-1 should listen to partition-1 for all the topics on different topologies?

Upvotes: 1

Views: 782

Answers (1)

nipuna
nipuna

Reputation: 4125

There is a consumer configuration called partition.assignment.strategy. There are Three options for this, Range, RoundRobin and StickyAssignor.

RangeAssignor

The RangeAssignor is the default strategy. The aims of this strategy is to co-localized partitions of several topics. This is useful, for example, to join records from two topics which have the same number of partitions and the same key-partitioning logic.

RoundRobinAssignor

The RoundRobinAssignor can be used to distribute available partitions evenly across all members. As previously, the assignor will put partitions and consumers in lexicographic order before assigning each partitions.

StickyAssignor

The StickyAssignor is pretty similar to the RoundRobin except that it will try to minimize partition movements between two assignments, all while ensuring a uniform distribution.

I think in your scenario, you should go with RangeAssignor. It makes sure that same consumer has same partition from different topics if they have same number of partitions.

you can find great documentation here

Upvotes: 1

Related Questions