CODEWITHSUNDEEP
hadoopmapreduce
sunillp
sunillp

Reputation: 993

Hadoop per-file block size

In Hadoop book it is said that we can specify per-file block size at the time of creation of file.

"The most natural way to increase the split size is to have larger blocks in HDFS, by setting dfs.block.size, or on a per-file basis at file construction time."

Any idea how to do this at file construction time. I hope by setting this to value = file-size, the file will not be split

Upvotes: 5

Views: 2889

Answers (1)

owen wang
owen wang

Reputation: 96

you can use CLI:

hadoop fs -D dfs.block.size=file-size -put local_name remote_location

or you can use Java API to specify the dfs.block.size when you want to create or copy files.

Configuration conf = new Configuration();
conf.setInt("dfs.block.size",file-size);

Upvotes: 8

Related Questions