CODEWITHSUNDEEP
apache-kafkaapache-kafka-streams
posthumecaver
posthumecaver

Reputation: 1863

Kafka Streams / Reduce and log retention

I have a question about log retention and Kafka Streams reduce operations...

I have a following stream configuration...

builder.
 .stream("topic1", Comsumed.with(Serdes.string(), Serdes.string()))
 .groupByKey(Serialized.with(Serdes.string(), Serdes.string()))
.reduce((val1, val2) -> val2, Materialzed.as("store1"))

I know tombstone events are not reaching the reduce function, so I have to use special object signalling the delete but how it works if the log retention deletes a message, how can transfer this to MaterializedView?

Upvotes: 0

Views: 505

Answers (1)

Matthias J. Sax
Matthias J. Sax

Reputation: 62350

If you read a topic as a stream, and log retention truncates the topic, you cannot to get informed about this.

Personal note: I think what you are doing is an anti-pattern. And the issues you are hitting, is an indicator for this.

The only way to get "delete" events from a topic is, if you use a compacted topic and read it as a KTable. Deletes there only happens via tombstones that the KTable will receive and process those accordingly.

Upvotes: 1

Related Questions