CODEWITHSUNDEEP
apache-kafkastreaming
Arun Y
Arun Y

Reputation: 939

Kafka Streaming application reading only latest message after connection with Kafka

We are using Kafka Streaming library to build real time notification kind of system for incoming messages on a Kafka topic, so while the streaming app is running, it processes all incoming messages in a topic at real time and send notification if it encounter certain kind of pre-defined incoming message.

If in case the Streaming App is down and it is started again, we require to process only recent messages arriving after streaming app is initialized. This is to avoid processing old records which were not processed while streaming app was not running or down. By default the streaming App starts processing old messages since last committed offset. Is there any setting in Kafka Streaming App to allow processing only most recent message?

Upvotes: 2

Views: 4245

Answers (2)

Saïd Bouras
Saïd Bouras

Reputation: 296

Your assumption is correct. Even if your set auto.offset.reset to latest, your app already have a consumer offset.

So you will have to reset the offsets to latest with the kafka-consumer-groups command with those options --reset-offsets --to-latest --execute.

Check the different reset scenarios , you can even reset to a particularly datetime, or by period, from a file etc..

Upvotes: 1

bistros
bistros

Reputation: 1179

KafkaConsumer's 'auto.offset.reset' default value is 'latest' but You want to use KafkaStreams, default is 'earliest' reference : https://github.com/apache/kafka/blob/trunk/streams/src/main/java/org/apache/kafka/streams/StreamsConfig.java#L634

Therefore, if set auto.offset.reset is 'latest' it will be what you want.

Upvotes: 1

Related Questions