zwush
zwush

Reputation: 115

Can Kafka producer do consumer lag aware partitioning

We a Kafka producer produces messages to a topic that is consumed by only one group of consumers. The speed of consumers is quite different. We always have a certain amount of consumer lags as a buffer.

Is it possible for the producer to know the lag of each partition of the consumer group and produce messages to the partition with the lowest lag first? And is it possible if I use C/C++ clients or Java clients?

Thanks.

Upvotes: 2

Views: 952

Answers (2)

OneCricketeer
OneCricketeer

Reputation: 191743

It is possible, given that you can lookup any consumer group name's lag from within the producer, however as the consumers are running those values will fluctuate and you're coupling yourself to knowing which consumers are running, which would be an anti-pattern in Kafka.

I'd suggest tweaking other consumer settings like poll sizes / frequency

Upvotes: 2

Javier Gonzalez Benito
Javier Gonzalez Benito

Reputation: 362

Its not possible.

You must ensure that the policy you use distributes the work to all partitions equally. You can provide a custom partitioner that suits your needs.

Look for example: https://dzone.com/articles/custom-partitioner-in-kafka-lets-take-quick-tour.

Default partitioner strategy is round robin

Upvotes: 0

Related Questions