CODEWITHSUNDEEP
apache-kafkaapache-kafka-streams
hermanjakobsen
hermanjakobsen

Reputation: 151

Kafka Streams Create Time vs Log-Append Time

If the Kafka broker is configured with Log-Append Time, will the broker overwrite the message timestamp regardless of the timestamp set by the producer/stream?

From streams-time - Documentation, I get the impression that CreateTime is somewhat unreliable - the timestamp is sometimes only forwarded (e.g. when using context.forward()) and sometimes overwritten (e.g. when using puntuate()). I am therefore wondering if the Log-Append Time takes precedence when setting the message timestamp (i.e. overwrites the timestamp set by the stream)?

Upvotes: 3

Views: 4192

Answers (2)

Gilles Philippart
Gilles Philippart

Reputation: 645

The javadoc of the ProducerRecord class confirms that LogAppendTime takes precedence:

If the topic is configured to use LogAppendTime, the timestamp in the producer record will be overwritten by the broker with the broker local time when it appends the message to its log.

Upvotes: 1

Lucas Brutschy
Lucas Brutschy

Reputation: 730

Yes, LogAppendTime takes precedence, meaning if the topic is configured to use LogAppendTime, the timestamp set by the Streams application (no matter how it is generated) will be ignored and overwritten with the log append time by the broker when the output record is produced.

Furthermore, I don't think CreateTime is unreliable in the way you think. punctuate() doesn't overwrite an existing timestamp - it is used to generate a fresh record at a specific point in time, so there is no previous CreateTime available - Streams must pick a new timestamp.

Upvotes: 6

Related Questions