CODEWITHSUNDEEP
springapache-kafkakafka-consumer-apispring-kafka
mohsenJsh
mohsenJsh

Reputation: 2108

Number of Kafka consumer's clients decrease after rebalancing

I've noticed that after a period of time -for example two days- consumergroup concurrency become lower that one I config.

I use spring boot and here is my code sample

factory.setConcurrency(10);

when I use following kafka command after stating kafka consumer it show exactly 10 different consumer client

bin/kafka-consumer-groups.sh --describe --group samplaConsumer --bootstrap-server localhost:9092

after a period of time when I run upper command consumer clients become lower, for example 6 distinct client and manage those 10 partitions.

how can I fix this so after re-balancing or whatever number of clients remain constant

Upvotes: 1

Views: 163

Answers (1)

mohsenJsh
mohsenJsh

Reputation: 2108

I find out that if a consumer client takes more time than max.poll.interval.ms to process the polled data the consumer considers failed and the group will rebalance.

max.poll.interval.ms The maximum delay between invocations of poll() when using consumer group management. This places an upper bound on the amount of time that the consumer can be idle before fetching more records. If poll() is not called before the expiration of this timeout, then the consumer is considered failed and the group will rebalance in order to reassign the partitions to another member.

And I found out that if this happens a lot that consumer client considers dead and no more rebalancing happens so the number of consumer concurrent client will decrease.

One solution that I came into is that I can decrease the number of max.poll.records so that processing of records took less time than max.poll.interval.ms.

props.put(ConsumerConfig.MAX_POLL_RECORDS_CONFIG, 50); // default is 200

Upvotes: 1

Related Questions