CODEWITHSUNDEEP
javaapache-kafkaapache-kafka-streams
Zana Simsek
Zana Simsek

Reputation: 99

How can I change the consumer offset of Kafka Topology?

Nowadays, I'm working on a project that uses Kafka Streams to process messages. Our messages consists from two identifier, one of them for identifier of user and second for the identifier of message list which user belong.

The story of the application is that A client want to send push messages to the their users list via my application. So, the client loads the list of users from an external source before he send push messages to the user and after, starts sending. But the client also want to stop an ongoing sending whenever he want.

My problem arose at this point. I want to stop sending push messages without consuming all messages from kafka via its stream api. I googled for the change the offset of the stream's source as latest but cannot find a proper way to accomplish it.

How can I accomplish it? Thanks.

Upvotes: 1

Views: 786

Answers (1)

Matthias J. Sax
Matthias J. Sax

Reputation: 62310

Kafka Streams does not support this. If you really need to do this, you can stop Kafka Streams and use bin/kafka-consumer-groups.sh to manually seek-to-end for the Kafka Streams application (note: application.id == group.id). Afterwards you can restart Kafka Streams.

Btw: in upcoming 1.1 release, bin/kafka-streams-application-reset.sh will also allow to manipulate offsets. And I hope that we also manage to make the internally used class StreamsResetter public API -- this would allow you do to this programmatically within our application without the need to fall back to command line.

Upvotes: 2

Related Questions