CODEWITHSUNDEEP
apache-kafka
karthikraja
karthikraja

Reputation: 497

Migrate Zookeeper Offset details to Kafka

We were using previous version of Kafka(0.8.X) where all the offset details were stored in ZooKeeper. Now we moved to new version of Kafka(0.10.X) where all the Topic offset details are stored in Kafka itself. We have to move all the Topic offset details to ZooKeeper to Kafka for existing application in Production. Kafka is installed in Windows machine. we can't run kafka-consumer-groups.sh from windows. Please advice how to migrate committed offsets from ZooKeeper to Kafka.

Upvotes: 0

Views: 1001

Answers (1)

Dmitry Minkovsky
Dmitry Minkovsky

Reputation: 38193

From https://kafka.apache.org/documentation/#offsetmigration:

Kafka consumers in earlier releases store their offsets by default in ZooKeeper. It is possible to migrate these consumers to commit offsets into Kafka by following these steps:

  • Set offsets.storage=kafka and dual.commit.enabled=true in your consumer config.
  • Do a rolling bounce of your consumers and then verify that your consumers are healthy.
  • Set dual.commit.enabled=false in your consumer config.
  • Do a rolling bounce of your consumers and then verify that your consumers are healthy.
  • A roll-back (i.e., migrating from Kafka back to ZooKeeper) can also be performed using the above steps if you set offsets.storage=zookeeper.

My guessing is a "rolling bounce" means rolling restart. Not sure what else it could mean, and it makes sense in this context.

Upvotes: 1

Related Questions