CODEWITHSUNDEEP
apache-kafkaflumeflume-ng
wayuki
wayuki

Reputation: 83

how to send key value to kafka via Flume

I can send messages to kafka via Flume,but how to send key value messages to kafka ?

such as I use tailf log file as source,how can I send key value messages?

# Describe/configure the source
a1.sources.r1.type = exec
a1.sources.r1.command =tail -F /apps/logs/mysql_status.log

Upvotes: 0

Views: 510

Answers (1)

OneCricketeer
OneCricketeer

Reputation: 191738

You'll have to use Flume headers

Kafka Sink uses the topic and key properties from the FlumeEvent headers to send events to Kafka. If topic exists in the headers, the event will be sent to that specific topic, overriding the topic configured for the Sink. If key exists in the headers, the key will used by Kafka to partition the data between the topic partitions

Headers can be added with interceptors

.http://flume.apache.org/releases/content/1.9.0/FlumeUserGuide.html#flume-interceptors

Upvotes: 1

Related Questions