How does HDFS manage block size?

Question

My file size is 65MB and default hdfs block size(64MB), then how many 64MB blocks will be allotted to my file?

Is it like 1-64MB block, 1-1MB block or 2-64MB blocks? If it is 2-64MB blocks is it going to be wasted rest of the 63MB or will it be allocated to other file?

Answer 1

The answer is 2 blocks, one 64MB and other 1MB.

HDFS just like other filesystems splits the file into blocks and then saves those blocks to disks.

But there are two major differences between them :

1. HDFS block sizes are huge because every block has a metadata record at namenode, smaller block sizes means a lot of blocks and overloading of namenode with metadata.

Hence, bigger block sizes used in HDFS.

2. HDFS block sizes are just an abstraction on the linux based file system, hence 65MB will use one 64MB block and other 1MB space from second block, rest 63MB from second block is still free and available for other data.

That is, Namenode will have two blocks recorded for 65MB but the actual file system space is 65MB only.

Answer 2

Block size 64MB means an upper bound size for a block. It doesn't mean that file blocks less than 64MB will consume 64MB. It will not consume 64MB to store a chunk of 1MB.

If the file is 160 megabytes,

Hope this helps.

Answer 3

According to this page. Looks like it'll be one 64 MB block and one 1 MB block.

HDFS is often blissfully unaware that the final record in one block may be only a partial record, with the rest of its content shunted off to the following block. HDFS only wants to make sure that files are split into evenly sized blocks that match the predefined block size for the Hadoop instance... Not every file you need to store is an exact multiple of your system’s block size, so the final data block for a file uses only as much space as is needed.