CODEWITHSUNDEEP
apache-kafkakafka-consumer-apispring-kafka
jafwatt
jafwatt

Reputation: 323

How to make a Kafka wildcard consumer in Java aware of a new topic sooner?

My project is implementing multi-tenancy at the Kafka level by using a separate topic for each tenant and wildcard consumers, i.e. publish to topic 'message.tenant1' or 'message.tenant99' and consume from topic 'message.*'

This works fine until we want to dynamically add a new tenant, i.e. add the 'message.tenant100' topic. The wildcard consumer won't see the new topic until it's been restarted.

Is there a way of making wildcard consumers see new topics without restarting the entire application?

We're using Spring but if the solution isn't available through Spring then we can use something else.

EDIT: Turns out this does work but there's a rough;y 5 minute delay before a re-balance occurs. 5 minutes is likely to be too long for us in production. I tried setting 'leader.imbalance.check.interval.seconds' to a lower value but this didn't seem to have any effect.

How do I either configure or tell Kafka to re-balance sooner? I expect re-balancing is an expensive operation and not one you want to do too often.

Upvotes: 2

Views: 2780

Answers (1)

mjuarez
mjuarez

Reputation: 16824

There's a consumer property, metadata.max.age.ms that can force consumers to refresh their metadata. The default is 300000, or 5 minutes. You could change this to every 30 seconds or so, to have a shorter delay to recognize new topics.

From the official Kafka docs, under New Consumer Configs:

metadata.max.age.ms - The period of time in milliseconds after which we force a refresh of metadata even if we haven't seen any partition leadership changes to proactively discover any new brokers or partitions.

https://kafka.apache.org/documentation/#newconsumerconfigs

Upvotes: 8

Related Questions