CODEWITHSUNDEEP
hadoop
Debbroto Sarker Debu
Debbroto Sarker Debu

Reputation: 11

Data replication in hadoop cluster

I am a beginner learning Hadoop. Is it possible that 2 different data blocks from the same file could be stored in the same data node? For example: blk-A and blk-B from file "file.txt" could be placed in the same data node (datanode 1).

Upvotes: 0

Views: 1226

Answers (2)

RojoSam
RojoSam

Reputation: 1496

The answer depends on the cluster topology. Hadoop tries to distribute data among data centers and data nodes. But What if you only have one data center ? or if you have only one node cluster (pseudo cluster). In those cases the optimal distribution doesn't happen and it is possible that all blocks end in the same data node. In production it is recommended have more than one data center (physically, not only in configuration) and at least the same number of data nodes than the replication number.

Upvotes: 0

sam
sam

Reputation: 2033

Here is the documentation that explains block placement policy. Currently, HDFS replication is 3 by default which means there are 3 replicas of a block. The way they are placed is:

  1. One block is placed on a datanode on a unique rack.
  2. Second block is placed on a datanode on a different rack.
  3. Third block is placed on a different datanode on the same rack as second block.

This policy helps when there is an event such as datanode is dead, block gets corrupted, etc.

Is it possible?

Unless you make changes in the source code, there is no property that you can change that will allow you to place two blocks on same datanode.

My opinion is that placing two blocks on same datanode beats the purpose of HDFS. Blocks are replicated so HDFS can recover for reasons described above. If blocks are placed on same datanode and that datanode is dead, you will lose two blocks instead of one.

Upvotes: 1

Related Questions