CODEWITHSUNDEEP
apache-kafka-streamsrocksdb
Chris
Chris

Reputation: 1349

Kafka Streams Window Store Retain Duplicates

The docs are unclear. When would I want to set retain duplicates to false/true. What is this used for? Is it for something particular in RocksDB?

https://kafka.apache.org/21/javadoc/org/apache/kafka/streams/state/Stores.html#persistentWindowStore-java.lang.String-java.time.Duration-java.time.Duration-boolean-

Digging through streams internal code seems to being used to set some sequence number?

RocksDBWindowStore.java

private void maybeUpdateSeqnumForDups() {
    if (this.retainDuplicates) {
      this.seqnum = this.seqnum + 1 & 2147483647;
    }

Upvotes: 5

Views: 1532

Answers (1)

Matthias J. Sax
Matthias J. Sax

Reputation: 62330

Well, as the name indicates, you can enable storing duplicates if you want to store multiple rows, with the same key. For window stores, the key is comprise of record key and window start timestamp.

Kafka Streams uses this feature for KStream-KStream joins. For this case, each input record is stored in its own window in the store (using the record timestamp as window start timestamp). Because there might be multiple records with the same key and same timestamp, it's required to enable this flag to compute the correct join. Otherwise, the join result might be incomplete.

Upvotes: 4

Related Questions