CODEWITHSUNDEEP
apache-kafkaflume
John_Doe
John_Doe

Reputation: 37

How to send key, value messages with flume to a kafka producer

In console you add producer properties --property "parse.key=true" --property "key.separator=:" to produce key-value data into Kafka, but how to do this with flume? I tried to add

a1.sinks.k1.producer.parse.key=true 
a1.sinks.k1.producer.key.separator=:

in .conf file but was of no avail, the kafka treated the key like a string.

Upvotes: 0

Views: 288

Answers (1)

OneCricketeer
OneCricketeer

Reputation: 191743

Those are console-producer CLI arguments, not ProducerConfig properties for Kafka (which are passed to Flume)

The key will always be a string, but you pass it via the headers of the Flume record

https://github.com/apache/flume/blob/trunk/flume-ng-sinks/flume-ng-kafka-sink/src/main/java/org/apache/flume/sink/kafka/KafkaSink.java#L193

Upvotes: 1

Related Questions