CODEWITHSUNDEEP
javahadoopmapreduceconfiguration-fileshadoop-yarn
KD157
KD157

Reputation: 823

Mapreduce Job Configuration File Location

Where do i find job configuration files for a java mapreduce job which is run already, like the overridden default settings and other job specific settings. I am using hadoop 2.6.0. I assume there will be a job configuration file to every job that is run with its user settings. Sorry if this question is repeated was not able to find the exact answer. There is nothing related to this in my yarn-site.xml or mapred-site.xml.

Upvotes: 1

Views: 3422

Answers (1)

Manjunath Ballur
Manjunath Ballur

Reputation: 6343

In your mapred-site.xml, there is configuration parameter: yarn.app.mapreduce.am.staging-dir.

The description of this parameter is:

The staging dir used while submitting jobs.

By default, this is set to:

/tmp/hadoop-yarn/staging

This path gets translated into:

{Value of config parameter `yarn.app.mapreduce.am.staging-dir`} + Path.SEPARATOR + user + Path.SEPARATOR + STAGING_CONSTANT

For e.g. in my mapred-site.xml, this value is set to:

<property>
    <name>yarn.app.mapreduce.am.staging-dir</name>
    <value>/user</value>
 </property>

So this path gets converted into:

/user/{user}/{STAGING_CONSTANT}

which is:

/user/mballur/.staging/ => Where user -> mballur and STAGING_CONSTANT -> .staging

In this folder, you will see your job related settings, when your job is running.

For e.g. if my application ID is: application_1450100618247_0018, then the staging folder will be:

/user/mballur/.staging/job_1450100618247_0018/

In this folder, you can see following files:

job.jar
job.split
job.xml
job_1450100618247_0018_1.jhist
job_1450100618247_0018_1_conf.xml

Here job.xml contains configuration information about your job. But, when the job is completed (succeeded or failed), the job's staging directory gets deleted.

Job History Rest API:

If you have enabled the History Server, then you can view the history of all the jobs executed so far. For e.g. check the link here: https://hadoop.apache.org/docs/r2.4.1/hadoop-yarn/hadoop-yarn-site/HistoryServerRest.html#Job_Conf_API

Using the following REST query, you can get the job's configuration:

GET http://<history server http address:port>/ws/v1/history/mapreduce/jobs/{job_id}/conf

Upvotes: 1

Related Questions