CODEWITHSUNDEEP
apache-kafkaapache-kafka-streams
vineet gandhi
vineet gandhi

Reputation: 87

Kafka Streams conditional topology

I want all records in topic say "B.topic" to be processed first and then when all stream threads are sitting idle then switch to primary topic "topic" and start consuming records.

This use case is mainly coming from disaster recovery perspective where in case of any DR MM2 is copying to remote "B.topic" and that needs to be consumed first to maintain order(may not be exact) and then consume from "topic".

I tried creating two topologies with one subscribed to "B.topic" and other to "topic". Once my application starts in DR region, it will first poll records from "B.topic" if no records are present then switch topology and continue consuming from "topic". Now how to know there are no left records in "B.topic" out of box from Kafka streams ? As I see the stream threads state are in running only.

Upvotes: 0

Views: 354

Answers (1)

Matthias J. Sax
Matthias J. Sax

Reputation: 62330

Now how to know there are no left records in "B.topic" out of box from Kafka streams ? As I see the stream threads state are in running only.

The state of KafakStreams will remain in RUNNING as it will continue to poll() for new data.

If you know that there won't be any new data, the lag for the consumer group (ie, application.id) should drop to zero. So you could look at the committed offsets on "B.topic" to see if all input data was processed?

Upvotes: 2

Related Questions