Reputation: 225
I am trying to manually assign partitions to each consumer within a consumer group. It seems like when you add consumers in a group, however, Kafka (or at least, kafka-python) assumes that you want the group coordinator to do all assignments. Is there a way to still keep all the benefits of using a consumer group (specifically, autocommit) while being able to manually manage assigning partitions? Can I write a custom partition assignor?
I am using the kafka-python 1.3.3 library.
Upvotes: 3
Views: 4432
Reputation: 747
you can use assign
method to manually assign partitions to consumer.
for example:
partition = TopicPartition('myTopic', 0)
self.consumer = KafkaConsumer(...)
self.consumer.assign([partition])
Upvotes: 1
Reputation: 1876
Yes,Kafka consumer has a property "partition.assignment.strategy" , where you can specify your own custom Assignor , default is RangeAssignor
Auto commit is also property driven "enable.auto.commit"
Kafka consumer configs : https://kafka.apache.org/documentation/#consumerconfigs
Upvotes: 2