CODEWITHSUNDEEP
javahadoophdfsflume
Pratik
Pratik

Reputation: 51

flume HDFS sink custom file name

Is there any way to configure(dynamically through Event) file name in HDFS sink stored by flume ? HDFS sink provides following name configuration.

hdfs.filePrefix
hdfs.fileSuffix

Upvotes: 0

Views: 3015

Answers (1)

Ripon Nandi
Ripon Nandi

Reputation: 68

Yes. You can configure the file name in HDFS sink by the following parameters.

hdfs.filePrefix = <<FILE_PREFIX>>-%{host}-%d-%H-%m-%s
hdfs.fileSuffix = <<FILE_SUFFIX>>-%{host}-%d-%H-%m-%s

%{host} = "It will pick up the hostname of your machine"
%t = "It will provide unix time in milliseconds "
%d= "Date"
%H= "Hour"
%m= "minutes"
%s= "seconds"

Thus Flume Bucket writer opens threads to hdfs to write, by using the "Prifix/Suffix" we can dynamically create filename.

Upvotes: 1

Related Questions