CODEWITHSUNDEEP
apache-kafka
FrVaBe
FrVaBe

Reputation: 49341

generic delete of obsolete Kafka consumer offsets without knowing the topic names

in my system the list of subscribed topics of a Kafka consumer has changed.

Some topics have been added, others are no longer subscribed, and still others were subscribed before and are still subscribed now.

I want to delete the ConsumerGroup offsets of the topics that are no longer subscribed by the active consumer. Otherwise the lag will still be listed and incremented for these topics which leads to an incorrect statement in the monitoring

I know the --delete-offsets option of the kafka-consumer-groups CLI tool. But here I need to know the affected topics and list them all. Is there a generic approach to delete obsolete topics of an active consumer without knowing the topics names?

Thanks!

Upvotes: 0

Views: 761

Answers (1)

nipuna
nipuna

Reputation: 4105

You can Change __consumer_offsets topic's configurations and do it.

  1. Change cleanup.policy config to delete

    default cleanup.policy for __consumer_offsets is compacted and therefore old messages never deleted because there is no new messages to the same key. Because yo are listening to different topi partitions.

  2. change retention.ms to 86400000 (one day) or your preferred duration

    default retention.ms for topics is 604800000 (7 days). Reduce it to one day (preferred time) will delete past logs older than this time in cleanup.policy=delete topics.

Wait for retention time. It will delete old logs

  1. Change again cleanup.policy to compact in __consumer_offsets topic

    This is the most important step. Because otherwise if your consumers pause for some period, last consumed offset will delete from the __consumer_offsets topic and there is no trace.

To do this, You don't need to know about past topics or consumer group ids. But this is risky because all your consumer groups last consumed offsets get deleted.

If this is not cleaning your old logs, please check your segments. Because may be not rolled out segments. If there is no old segments you need to reduce segment.ms or segment.bytes to roll out segments frequently. And rollback these changes as well.

Upvotes: 2

Related Questions