CODEWITHSUNDEEP
javaapache-kafkakafka-consumer-api
FrVaBe
FrVaBe

Reputation: 49361

Load balance Kafka record consumption when using topic pattern

If two processes which uses the same consumer group (group.id) subscribe to the same topicPattern - will Kafka than load balance the resolved topics between the two processes?

In my case about 400 topics match the pattern and each topic has one partion. I wonder if I can load balance the consumption of the topics via different processes with the same consumer group but it seems that only one process handles all the topics. Is this because only partitions are load balanced - or maybe because a topic pattern is used by the subscription?

I made some deeper debugging. My test setup is not really two processes but two threads - I guess that should not matter. Debug printing the topic description outputs something like this (IPs are obfuscated):

(groupId=foo-bar-group, isSimpleConsumerGroup=false, 
  members=
   (memberId=KafkaTestApplication:KafkaTestApplication-1-20126d55-678d-46f5-bc73-3769db2c8901, 
     clientId=KafkaTestApplication:KafkaTestApplication-1, 
     host=/aa.bb.c.dd, 
     assignment=(topicPartitions= <all topics here>)),
   (memberId=KafkaTestApplication:KafkaTestApplication-2-8d34e81a-8f57-4d4e-bd9e-b56edc4e706c, 
     clientId=KafkaTestApplication:KafkaTestApplication-2,
     host=/aa.bb.c.dd, 
     assignment=(topicPartitions=)), 
     partitionAssignor=range, 
     state=Stable, 
     coordinator=aaa.bb.ccc.d:9092 (id: 9 rack: null)
)

I can see that I have two different members in the consumer group with two different clientIds. Nevertheless all topics are assigned to the first consumer and the second consumer has no topics assigned. I expected that both members get some topics. Both members poll every few seconds.

Upvotes: 0

Views: 403

Answers (2)

FrVaBe
FrVaBe

Reputation: 49361

I finally found out what to change to get the desired topic/partition balancing. I had to set the Consumer property

partition.assignment.strategy=org.apache.kafka.clients.consumer.RoundRobinAssignor

The default settig (org.apache.kafka.clients.consumer.RangeAssignor) does not work for my setup. Why is explained here as user152468 mentioned in the comments. Thanks for that!

Upvotes: 0

OneCricketeer
OneCricketeer

Reputation: 192023

When the second process of the same group starts, the group should rebalance the Topic partition assignment, yes.

However, I've not had any experience with subscription patterns, but I don't expect it to work differently than providing a direct list of topics

Upvotes: 1

Related Questions