CODEWITHSUNDEEP
spring-integrationspring-cloud-stream
RSM
RSM

Reputation: 433

How to assign a specific kafka consumer to a specific partition

I have a "topic" with 2 partitions. 

topic-0
topic-1

And I have 2 input message channels 

input0
input1

with same destination "topic" and same group.

My understanding is that, since it is a same group each topic will be automatically assigned a partition.

Is possible to assign a specific partition to a specific message channel?

Upvotes: 4

Views: 1406

Answers (1)

Ilayaperumal Gopinathan
Ilayaperumal Gopinathan

Reputation: 4179

You can assign specific partitions to Kafka consumer with consumer's auto-rebalance disabled along with the appropriate instanceCount and instanceIndex properties.

For instance, in your case,

--spring.cloud.stream.bindings.input0.consumer.instanceCount=2 --spring.cloud.stream.bindings.input1.consumer.instanceCount=2 --spring.cloud.stream.bindings.input0.consumer.instanceIndex=0 --spring.cloud.stream.bindings.input1.consumer.instanceIndex=1 --spring.cloud.stream.bindings.input0.group=mygroup --spring.cloud.stream.bindings.input1.group=mygroup --spring.cloud.stream.kafka.bindings.input0.consumer.autoRebalanceEnabled=false --spring.cloud.stream.kafka.bindings.input1.consumer.autoRebalanceEnabled=false --spring.cloud.stream.bindings.input0.destination=topic --spring.cloud.stream.bindings.input1.destination=topic

The above configuration will assign the topic's partitions for each consumer (when binding the input channels) based on the modulo using the partition , instanceCount and instanceIndex values.

Upvotes: 3

Related Questions