CODEWITHSUNDEEP
apache-kafkaapache-kafka-streams
Pari
Pari

Reputation: 1493

Kafka Stream offset reset to zero for consumer group

I have written Kafka Streaming app that just filters rows based on some condition and load it to MongoDB.

The streaming process is working fine but due to some flaw in my code, I want to reprocess whole data again.

One way is kill streaming app , change consumer group id , delete data from mongo and rerun the app.

How to achieve this scenario without changing consumer group id.

<< I am using Kafka 0.10 version >>

Many Thanks Pari

Upvotes: 7

Views: 11496

Answers (2)

miguno
miguno

Reputation: 15087

Apache Kafka 0.10.0.1 (which was released in August, while the original question was asked in July) ships with a new Application Reset Tool for Kafka Streams, which is an easier and better/cleaner solution than simply renaming application.id.

You can execute the tool via the script bin/kafka-streams-application-reset.sh, which will also print a usage/help message.

Example:

# Run this only after ALL application instances were stopped!
$ bin/kafka-streams-application-reset --application-id my-streams-app \
                                      --input-topics my-input-topic \
                                      --intermediate-topics rekeyed-topic \
                                      --bootstrap-servers brokerHost:9092 \
                                      --zookeeper zookeeperHost:2181

That said, I'd recommend to read Data Reprocessing with Kafka Streams: Resetting a Streams Application, which the aforementioned Matthias J. Sax wrote, for further details. That article also explains why simply renaming application.id (which was the workaround until now) isn't the best idea.

Upvotes: 12

Pari
Pari

Reputation: 1493

Received update from Matthias J. Sax [email protected] -

currently, changing the application ID is the best way to follow. Cleaning up the application state correctly, is a little bit tricky. We are currently working on an improvement for this -- should be available soon.

See https://issues.apache.org/jira/browse/KAFKA-3185

Cheers Pari

Upvotes: 2

Related Questions