Hel
Hel

Reputation: 225

How do I manually assign partitions while still being able to auto commit?

I am trying to manually assign partitions to each consumer within a consumer group. It seems like when you add consumers in a group, however, Kafka (or at least, kafka-python) assumes that you want the group coordinator to do all assignments. Is there a way to still keep all the benefits of using a consumer group (specifically, autocommit) while being able to manually manage assigning partitions? Can I write a custom partition assignor?

I am using the kafka-python 1.3.3 library.

Upvotes: 3

Views: 4432

Answers (2)

Mohammad Sadegh
Mohammad Sadegh

Reputation: 747

you can use assign method to manually assign partitions to consumer. for example:

partition = TopicPartition('myTopic', 0)
self.consumer = KafkaConsumer(...)
self.consumer.assign([partition])

Upvotes: 1

Liju John
Liju John

Reputation: 1876

Yes,Kafka consumer has a property "partition.assignment.strategy" , where you can specify your own custom Assignor , default is RangeAssignor

Auto commit is also property driven "enable.auto.commit"

Kafka consumer configs : https://kafka.apache.org/documentation/#consumerconfigs

Upvotes: 2

Related Questions