CODEWITHSUNDEEP
hadoophdfs
parso
parso

Reputation: 91

Hadoop: Need to remove a single data directory from all data nodes

I need to reclaim disk space from all my Hadoop HDFS datanodes. They are each configured like this:

<property>
    <name>dfs.data.dir</name>
    <value>/d01,/d02,/d03</value>
</property>

How should I go about safely removing the /d03 filesystem from all nodes and rebalancing them? I've experimented with decommissioning a node, removing the directory, and recommissioning, but it is very slow and am wondering if there is a better way?

Upvotes: 4

Views: 3159

Answers (1)

parso
parso

Reputation: 91

The fix is actually very simple, and follows from knowledge of what HDFS is for. The filesystem is a distributed collection of replicated blocks that is failure tolerant. Therefore, simply removing the extra directories from a datanode in the cluster and restarting is sufficient to cause re-synchronisation and replication of the blocks to occur.

Monitor the NameNode log and the WebUI "Number of Under-Replicated Blocks" to discern when the process is complete.

Repeat for individually for all nodes in the cluster.

Several caveats:

Ensure that

  • there are no under-replicated blocks, by checking: http://<name-node>:50070,

  • there is adequate disk space across the other filesystems,

  • the replication level is set to at least 2 to ensure that the cluster can tolerate the loss of blocks. Ideally, this should be three or higher for safety.

Upvotes: 5

Related Questions