CODEWITHSUNDEEP
configurationhadoopclouderahadoop-yarn
Dolan Antenucci
Dolan Antenucci

Reputation: 15942

How do I view my Hadoop job history and logs using CDH4 and Yarn?

I downloaded the CDH4 tar for Hadoop with Yarn, and jobs are running fine, but I can't figure out where to view the logs from my job. In MRv1, I simply went to the JobTracker web app, and it had the job history. Individual jobs' logs were accessible from here as well, or by going to logs/userlogs directory.

In my new Yarn setup (just running on single computer), I have the logs directory, but no logs/userlogs folder.

When I go to the ResourceManager web page, localhost:8088, there is an "All Applications" section, but no entries show here. When I go to the JobHistory web page, localhost:19888, there are no jobs listed.

I'm wondering if there is a configuration issue. Here's my yarn-site.xml entries:

<property>
  <name>yarn.nodemanager.local-dirs</name> 
  <value>/Users/dolan/cdh4/_hadoop_data/yarn/local</value>
</property> 
<property> 
  <name>yarn.nodemanager.log-dirs</name>
  <value>/Users/dolan/cdh4/_hadoop_data/yarn/logs</value> 
</property>
<property> 
  <name>mapreduce.jobhistory.address</name>
  <value>localhost:10020</value> 
</property>
<property> 
  <name>mapreduce.jobhistory.webapp.address</name>
  <value>localhost:19888</value> 
</property>
<property>
  <name>yarn.app.mapreduce.am.staging-dir</name>
  <value>/user</value>
</property>

Any tips on what to debug are greatly appreciated.

Update: I've checked the nodemanager log dir, and it is empty. Additionally, the jobhistory directories are empty. One odd thing is that Hadoop tells me, INFO mapreduce.Job: The url to track the job: http://localhost:8080/, but that address isn't available when I try from browser.

Update #2: I noticed that each job I run has the same Job ID ("job_local_0001"). This seems like a configuration issue somewhere.

Upvotes: 4

Views: 20180

Answers (2)

mehikmat
mehikmat

Reputation: 80

Add following configuration into mapred-site.xml file

<property>
  <name> mapreduce.framework.name</name>
  <value>yarn</value>
</property>
<property> 
  <name>mapreduce.jobhistory.address</name>
  <value>localhost:10020</value> 
</property>
<property> 
  <name>mapreduce.jobhistory.webapp.address</name>
  <value>localhost:19888</value> 
</property>

and remove jobhistory specific configurations as you had put in yarn-site.xml eg. above last two configs from yarn-site.xml

Upvotes: 2

Sandy Ryza
Sandy Ryza

Reputation: 71

It sounds like the jobs here are being run locally and not on YARN at all. To make run the jobs on YARN, make sure you have this in your mapred-site.xml:

<property>
  <name> mapreduce.framework.name</name>
  <value>yarn</value>
</property>

In general, you'll probably get a faster response on questions like this by emailing the cdh-user mailing list.

Upvotes: 7

Related Questions