CODEWITHSUNDEEP
javaapache-kafkakafka-consumer-api
Milan Panic
Milan Panic

Reputation: 503

Consumer can't poll from newly created partition until it's restarted

I have a Spring Boot project that implements Kafka. The application uses producer to produce a message on a topic in a specific partition. The partitions on the topic are made dynamically on run time. After I add a new partition, the already running consumer for the given topic doesn't see this partition, but sees the older ones. However, when I restart the consumer, the consumer starts seeing the new partition and starts polling successfully from it.

My question is: Can I make this work without restarting the consumer and if I can, how? I can't seem to find this anywhere in the official documentation.

Partition creation:

 Map<String,NewPartitions> increaseTopicPartitions = new HashMap<String,NewPartitions>();
            increaseTopicPartitions.put("nodesTopic", NewPartitions.increaseTo(totalPartitions + 1));
            BaseProcessor.adminClient.createPartitions(increaseTopicPartitions);

Printing partition count:

 int totalPartitions = cluster.partitionCountForTopic(topic);

Upvotes: 1

Views: 75

Answers (1)

Mickael Maison
Mickael Maison

Reputation: 26885

The consumer will only see the new partitions when it refreshes its cluster metadata. By default this only happens every 5 mins (see metadata.max.age.ms in the Consumer configurations).

Once the new partitions are detected, they should be correctly assigned and the consumer should start receiving messages.

Upvotes: 3

Related Questions