CODEWITHSUNDEEP
apacheflume
Raghuveer
Raghuveer

Reputation: 3057

Flume generated files

I have a couple of questions

I have flume configuration as note here in answer

  1. Flume is generating a file with the name Flume-data.millisecond. Can I assign a custom name? How can I change it?

  2. With this configuration I was able to load a 1.5MB file 5 times but when I ran it for the 6th time I got an OutofMemoryError. How can I resolve this issue?

Thanks

Upvotes: 1

Views: 306

Answers (1)

Amar
Amar

Reputation: 3845

Flume is generating a file and Flume-data.millisecond is the name of the file, can i have a custom name? how can i change it?

Yes you can have a custom name as a prefix or suffix to the files being generated.

hdfs.filePrefix : FlumeData : Name prefixed to files created by Flume in hdfs directory

hdfs.fileSuffix – Suffix to append to file (eg .avro - NOTE: period is not automatically added)

Check out the HDFS sink in flume user guide for more details.

With this configuration i was able to load a 1.5MB file 5 times but when i ran it for the 6th time i got an OutofMemoryError. How can i resolve this issue.

You are getting out of memory error because you are using a memory channel with a insufficient capacity. Better using a File channel.

Upvotes: 1

Related Questions