How to load balance Kafka consumers within a consumer group

Question

Scenario

• 10 kafka consumers within a same Consumer Group.
• Kafka has 10 partitions => which means each partition is automatically assigned to a single consumer within the group.
• Message is sent to partition on a round-robin basis.
• Every now and then, a message will take much longer to process than other messages.
• In such occasions, there's a chance the next message is assigned to a consumer that is still busy working while there are other free consumers

Question

1. Does Kafka support a mechanism to automatically send message to a partition whose consumer is free?
2. If it doesn't, what is the common approach to this scenario?

Answer 1

Although you could implement a custom Assignor class, by default, consumption is only based on assignment, not by load; such information is not communicated back to the group coordinator. Plus, shuffling around constantly based on load would likely cause frequent group rebalances, causing consumption to be even slower

Regarding length-of-processing, I am not aware of any way your consumer would be able to inspect message before partition assignment and polling such records. Therefore, you'd need to decouple your processing logic from the actual poll loop if you'd like to improve processing times.