CODEWITHSUNDEEP
hadoop
Julias
Julias

Reputation: 5892

how to update config parameter of hadoop mapred-site.xml without restarting the cluster

I would like to add/update the following parameter 

 <property>
    <name>mapred.map.tasks.speculative.execution</name>
    <value>false</value>
  </property>

in mapred-site.xml in hadoop. How can I do this without restarting the cluster?

Upvotes: 3

Views: 7026

Answers (3)

Vijayanand
Vijayanand

Reputation: 500

If you are using MRv2 (YARN for Resource scheduling), the changes made in mapred-site.xml file will be read every time when you submit the job and all the configs will be saved as job config xml file (when the job is submitted to cluster). There is no need to restart any services.

Upvotes: 2

Praveen Sripati
Praveen Sripati

Reputation: 33545

According to the Hadoop - The Definitive Guide

Be aware that some properties have no effect when set in the client configuration. For example, if you set mapred.tasktracker.map.tasks.maximum in your job submission with the expectation that it would change the number of task slots for the tasktrackers running your job, you would be disappointed, because this property is honored only if set in the tasktracker’s mapred-site.xml file. In general, you can tell the component where a property should be set by its name, so the fact that mapred.tasktracker.map.tasks.maximum starts with mapred.tasktracker gives you a clue that it can be set only for the tasktracker daemon. This is not a hard and fast rule, however, so in some cases you may need to resort to trial and error, or even to reading the source.

The properties which are honored in the client side and are job specific don't a need a cluster restart, but on the other hand cluster specific configurations like mapred.tasktracker.map.tasks.maximum would require the cluster to be restarted.

Upvotes: 5

Thomas Jungblut
Thomas Jungblut

Reputation: 20969

You can set this per-job in your job's configuration. If you want to change this behaviour globally you have to restart your cluster. There is no way to change it without restarting.

Upvotes: 6

Related Questions