CODEWITHSUNDEEP
hadoopapache-kafkahdfsflume
philip78yahoo
philip78yahoo

Reputation: 9

Use of flume as a kafka consumer

Is it possible to config flume sink to be my agent's file system. Do I have to sink to hdfs or hadoop?
I am working with flume 1.6.0 and kafka 10.1.1
I will show you my flume config and flume command line args if you ask but maybe I'm doing something that is just not meant to be done.
I am trying to do some proof-of-concept on the kafka side without installing hadoop or hdfs.
I see config for roll_file but maybe in these versions such a concept is for hdfs only?

Upvotes: 0

Views: 384

Answers (2)

Kiran Balakrishnan
Kiran Balakrishnan

Reputation: 257

if you want to use flume,you have to create a flume agent from ambari or cloudera manager which ever you are using.You will have to have hdfs to sink the data from kafka. Source will be kafka topic Channel can be mem Sink HDFS

Upvotes: 0

OneCricketeer
OneCricketeer

Reputation: 191681

File Roll Sink says

Stores events on the local filesystem

However, I would suggest not using Flume as it requires you to install extra Hadoop libraries.

Kafka Connect is a native Kafka library and you can consume to a File (or HDFS).

Upvotes: 2

Related Questions