CODEWITHSUNDEEP
hadoophdfs
user13128577
user13128577

Reputation:

How does Hadoop HDFS decide what data to be put into each block?

I have been trying to dive into how Hadoop HDFS decides what data to be put into one block and don't seem to find any solid answer. We know that Hadoop will automatically distribute data into blocks in HDFS across the cluster, however what data of each file should be put together in a block? Will it just put it arbitrarily ? And is this the same for Spark RDD?

Upvotes: 0

Views: 150

Answers (1)

sathya
sathya

Reputation: 2072

  1. HDFS block behavior

I'll attempt to highlight by way of example the differences in blocks splits in reference to file size. In HDFS you have:

Splittable FileA size 1GB
dfs.block.size=67108864(~64MB)

MapRed job against this file:

16 splits and in turn 16 mappers.

Let's look at this scenario with a compressed (non-splittable) file:

Non-Splittable FileA.gzip size 1GB
dfs.block.size=67108864(~64MB)

MapRed job against this file:

16 Blocks will converge on 1 mapper.

It's best to proactively avoid this situation since it means that the tasktracker will have to fetch 16 blocks of data most of which will not be local to the tasktracker.

  1. spark reading a HDFS splittable file:

sc.textFile doesn't commence any reading. It simply defines a driver-resident data structure which can be used for further processing.

It is not until an action is called on an RDD that Spark will build up a strategy to perform all the required transforms (including the read) and then return the result.

If there is an action called to run the sequence, and your next transformation after the read is to map, then Spark will need to read a small section of lines of the file (according to the partitioning strategy based on the number of cores) and then immediately start to map it until it needs to return a result to the driver, or shuffle before the next sequence of transformations.

If your partitioning strategy (defaultMinPartitions) seems to be swamping the workers because the java representation of your partition (an InputSplit in HDFS terms) is bigger than available executor memory, then you need to specify the number of partitions to read as the second parameter to textFile. You can calculate the ideal number of partitions by dividing your file size by your target partition size (allowing for memory growth). A simple check that the file can be read would be:

    sc.textFile(file, numPartitions)
      .count()

Upvotes: 0

Related Questions