CODEWITHSUNDEEP
apache-kafkaspring-kafka
J.R.Smith
J.R.Smith

Reputation: 1

Kafka remove unused consumer groups

I have an application that uses Apache Kafka and creates a new consumer group on every startup. It takes fixed string and adds generated uuid to generate group id. (ex. my_consumer_group_123324234234, my_consumer_group_123324234235 ...). When I shut down the app old consumer groups stays unused until offsets.retention.minutes after kafka doesn't remove them. I wonder if it is possible to remove unused consumer groups (filtered with name like 'my_consumer_group_*') by script

Upvotes: 0

Views: 2542

Answers (1)

Gerard Garcia
Gerard Garcia

Reputation: 1856

Yes, it should be possible using the kafka-consumer-groups.sh script included with Kafka.

You could create a script that periodically lists the existing consumer groups

kafka-consumer-groups.sh --bootstrap-server <kafka-servers-addrs> --list

Then describes each one of them

kafka-consumer-groups.sh --bootstrap-server <kafka-servers-addrs> --describe --group <consumer-group>

One option to detect if they are unused is to parse the output to see if it returns:

Consumer group '<consumer-group>' has no active members.

Note that relying on the message could be a bit brittle, since the message could change across Kafka versions, so I'd look for some other more robust approach (e.g. status code that the script returns (if any), initialize your own consumer...)

And then deletes the ones that are unused:

kafka-consumer-groups.sh --bootstrap-server <kafka-server-addrs> --delete --group <consumer-group1> --group <consumer-group2>

Upvotes: 2

Related Questions