CODEWITHSUNDEEP
apache-kafkalogstash
eightnoteight
eightnoteight

Reputation: 234

How to seperate a consumer from consumer group without losing offset?

I have a logstash kafka consumer group subscribing nearly 20topics which isn't performing well for a particular high priority kafka topic to process, so I've decided to remove one topic out of the consumer group and launch a separate consumer group for high priority topics, but unfortunately with that i'm losing the offset that was there in old consumer group.

Is there anyway I can start the new logstash consumer group with the initial offset from last consumer group?

Thanks

Upvotes: 0

Views: 139

Answers (1)

Bartosz Wardziński
Bartosz Wardziński

Reputation: 6603

You can use kafka scripts to set offset for new group.

Sample scenario:

  1. Stop application.

  2. Check current offset for group. You can use following command. The output will contains information about current offset, log end offset, lag, etc for each topic.

    ./bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group groupId --describe

  3. Set offset for new group id, that will be used by new application (suppose the offset for topic you would like to switch is 10001)

    ./bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group newGroupId --to-offset 10001 --topic topicName --reset-offsets --execute

  4. Remove topicName from topics list for old application.

  5. Set up new logstash configuration with newGroupId group id.
  6. Start old and new logstash applications.

Upvotes: 1

Related Questions