CODEWITHSUNDEEP
hadoophdfs
sunil kancharlapalli
sunil kancharlapalli

Reputation: 105

Writing a file larger than block size in hdfs

If I am trying to write a file of 200MB into HDFS where HDFS block size is 128MB. What happens if the write fails after writing 150MB out of 200MB. Will I be able to read data from the portion of data written? What if I try to write the same file again? Will that be a duplicate? What happens to the 150MB of data written earlier to failure?

Upvotes: 2

Views: 562

Answers (1)

andani
andani

Reputation: 424

HDFS default Block Size is 128MB, if it fails while writing (it will show the status in Hadoop Administration UI, with file extension copying.)

  • Only 150MB data will be copied.

  • yeah you can read only portion of data(150MB).

  • Once you reinstate the copying it will continue from previous point(if both the paths are same and file name is same).

  • For every piece of data you can find the replication based on your replication factor.

  • Previous written data will be available in HDFS.

Upvotes: 3

Related Questions