CODEWITHSUNDEEP
hadoophdfs
alekya reddy
alekya reddy

Reputation: 934

Datanode decommisioning vs Write

What happens when we decommission a datanode while write is happening to HDFS on that node?

Will it stop writing the data to HDFS on that node and decommision that node or will it finish writing and then decommision it.

Upvotes: 3

Views: 448

Answers (3)

alekya reddy
alekya reddy

Reputation: 934

I found a solution for this in cloudera.

Decommissioning applies to only to HDFS DataNode, MapReduce TaskTracker, YARN NodeManager, and HBase RegionServer roles. If the host has other roles running on it, those roles are stopped/killed.

http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/cm_mc_decomm_host.html

Upvotes: 1

Karthik
Karthik

Reputation: 1811

After you add the entries for the nodes that you want to decommission. You need to execute namenode -refreshnodes command to start actual process of decommission. Actual process of decomission is very slow. All the current tasks either read/write will continue as per usual. But all future writes are not allowed but reads are allowed with least priority. Meanwhile all the blocks are replicated to other nodes.

Upvotes: 0

frb
frb

Reputation: 3798

As staded at the Hadoop wiki:

Hadoop offers the decommission feature to retire a set of existing data-nodes. The nodes to be retired should be included into the exclude file, and the exclude file name should be specified as a configuration parameter dfs.hosts.exclude. This file should have been specified during namenode startup.

The exclude file property names a file that contains a list of hosts that are not permitted to connect to the namenode.

When that occurs, I think an IOException is gotten by the process writting to the decommissioned node.

Upvotes: 0

Related Questions