Jet Kafka Load Balancing

Question

With Jet 0.4, there was an option to load balance Jet instances reading from a Kafka Topic by using a common group id.

With current Jet 0.5, this group.id value is ignored, how do we prevent Jet instances from getting duplicate copy of the message? In other words, we want the workload (processing Kafka topic's messages) to be load balanced in the Jet instances, how do we do this with 0.5?

Answer 1

Jet 0.5 uses manual partition assignment. The partitions are split evenly among members even in this case, there won't be duplicates. There was an issue when it was using Kafka's partition assignment.