CODEWITHSUNDEEP
apache-kafkakafka-consumer-apikafka-producer-api
md1980
md1980

Reputation: 339

kafka header and key value pair

I came across this link for kafka headers. As discussed in the below link

kafka headerhttps://cwiki.apache.org/confluence/display/KAFKA/KIP-82+-+Add+Record+Headers

Can i use key/value pair in producer-api to send header and payload information? I see a point about log compaction, can i turn off log compaction so that in my case when every key will be unique, the logs won't keep track of every record?

i did create a producer with key value pair where the key had the header information and value is the payload. Every key is unique.

Consumer is also receiving the correct key value pair.

I would like to know if there will be any issues with using this approach for header information with log compaction on high volume. Is turning off log compation an option and let retention clean up older data?

Upvotes: 1

Views: 4464

Answers (2)

ppatierno
ppatierno

Reputation: 10065

Adding to what cricket_007 said and if I understood the question, the log compaction feature is not based on keys used inside the header but for the key in the Kafka message. Every Kafka message has a KEY, a payload and the headers (which are of course key/value pairs) and then more other things (timestamp, ...). The log compaction works on the KEY of the message and not on the keys in the headers map.

Upvotes: 1

OneCricketeer
OneCricketeer

Reputation: 191844

Headers don't control log compaction.

Yes, you can producer.send() a ProducerRecord with Headers

With log compaction on, or off, all records of unique keys within the retention window are kept, so not sure I understand that question

Upvotes: 1

Related Questions