Reputation: 1863
I have a question about log retention and Kafka Streams reduce operations...
I have a following stream configuration...
builder.
.stream("topic1", Comsumed.with(Serdes.string(), Serdes.string()))
.groupByKey(Serialized.with(Serdes.string(), Serdes.string()))
.reduce((val1, val2) -> val2, Materialzed.as("store1"))
I know tombstone events are not reaching the reduce function, so I have to use special object signalling the delete but how it works if the log retention deletes a message, how can transfer this to MaterializedView?
Upvotes: 0
Views: 505
Reputation: 62350
If you read a topic as a stream, and log retention truncates the topic, you cannot to get informed about this.
Personal note: I think what you are doing is an anti-pattern. And the issues you are hitting, is an indicator for this.
The only way to get "delete" events from a topic is, if you use a compacted topic and read it as a KTable
. Deletes there only happens via tombstones that the KTable
will receive and process those accordingly.
Upvotes: 1