Reputation: 115
We a Kafka producer produces messages to a topic that is consumed by only one group of consumers. The speed of consumers is quite different. We always have a certain amount of consumer lags as a buffer.
Is it possible for the producer to know the lag of each partition of the consumer group and produce messages to the partition with the lowest lag first? And is it possible if I use C/C++ clients or Java clients?
Thanks.
Upvotes: 2
Views: 952
Reputation: 191743
It is possible, given that you can lookup any consumer group name's lag from within the producer, however as the consumers are running those values will fluctuate and you're coupling yourself to knowing which consumers are running, which would be an anti-pattern in Kafka.
I'd suggest tweaking other consumer settings like poll sizes / frequency
Upvotes: 2
Reputation: 362
Its not possible.
You must ensure that the policy you use distributes the work to all partitions equally. You can provide a custom partitioner that suits your needs.
Look for example: https://dzone.com/articles/custom-partitioner-in-kafka-lets-take-quick-tour.
Default partitioner strategy is round robin
Upvotes: 0