CODEWITHSUNDEEP
hadoopconfigurationip
Liaoxiaochen
Liaoxiaochen

Reputation: 139

what would happen if nodes in hadoop change their IP address?

my hadoop clusters do not work fine because of the network conditions.What if i change the entire network,like another router,thus change the IP addresses? could the clusters still work by updating some configurations? or i must torn it down and rebuilt everything? Thanks in advance

Upvotes: 1

Views: 1141

Answers (1)

ozw1z5rd
ozw1z5rd

Reputation: 3208

It works once you change the ip addresses into the configuration, why did not you use the DNS?

Ok, it was not a good answer, let me apologize and give a better answer. If you need to change configuration on a running cluster you can decommission and commission the data nodes.

Switch off the data node is not a good idea.

Data Node Decomissioning

The fist step is tell to yarn you are going to remove some nodes, then you have to say the same to node manager.

I don't know if your system is configured for decommissioning, if it so you have the key yarn.resourcemanager.nodes.exclude-path into the yarn-site.xml and dfs.hosts.exclude into hdfs-site.xml

hdfs-site.xml 

<property>
  <name>dfs.hosts.exclude</name>
  <value>$YOUR_PATH/dfs.exclude</value>
  <final>true</final>
</property>

yarn-site.xml

<property>
  <name>dfs.hosts.exclude</name>
  <value>$YOUR_PATH/dfs.exclude</value>
  <final>true</final>
</property>

Open the file $YOUR_PATH/dfs.exclude and add hostnames / ip addresses of node you need to stop.

execute

yarn rmadmin  -refreshNodes
hdfs dfsadmin -refreshNodes

Check if the data nodes are in decommission checking the web interface.

Data Node Comissioning

Works in the same way of the Decommissioning

yarn-site.xml

<property>
  <name>yarn.resourcemanager.nodes.include-path</name>
  <value>$YOUR_PATH/dfs.include</value>
  <final>true</final>
</property>

hdfs-site.xml

<property>
  <name>dfs.hosts</name>
  <value>$YOUR_PATH/dfs.include</value>
  <final>true</final>
</property>

Open the file $YOUR_PATH/dfs.include and add hostnames / ip addresses of node you need to add.

yarn rmadmin  -refreshNodes
hdfs dfsadmin -refreshNodes

wait some time

 hdfs dfsadmin -report

Now the hosts you added are into the list.

If your configurations are missing the above keys you need to halt/restart the node manager and yarn after adding them.

Using these procedure you can halt data nodes in a safe way.

Upvotes: 1

Related Questions