CODEWITHSUNDEEP
hadooploggingmapreduce
pgrandjean
pgrandjean

Reputation: 766

How to collect Hadoop userlogs?

I am running M/R jobs and logging errors when they occur, rather than making the job fail. There are only a few errors, but the job is run on a hadoop cluster with hundreds of nodes. How to search in task logs without having to manually open each task log in the web ui (jobtaskhistory)? In other words, how to automatically search in M/R task logs that are spread all over the cluster, stored in each node locally?

Upvotes: 0

Views: 951

Answers (4)

pgrandjean
pgrandjean

Reputation: 766

yarn logs -applicationId <myAppId> | grep ...

Upvotes: 0

inor
inor

Reputation: 2846

Go to to your http://sandbox-hdp.hortonworks.com:8088/cluster/apps
There look for the instantiation of the execution you are interested in, and for that entry click the History link (in the Tracking UI column),
then look for the Logs link (in the Logs column), and click on it

Upvotes: 0

Pradeep Gollakota
Pradeep Gollakota

Reputation: 2181

Side Note First: 2.0.0 is oldy moldy (that's the "beta" version of 2.0), you should consider upgrading to a newer stack (e.g. 2.4, 2.5 2.6).

Starting with 2.0, Hadoop implemented what's called "log aggregation" (though it's not what you would think. The logs are just stored on HDFS). There are bunch of command line tools that you can use to get the logs and analyze them without having to go through the UI. This is, in fact, much faster than the UI.

Check out this blog post for more information.

Unfortunately, even with the command line tool, there's not way for you to get all task logs at the same time and pipe it to something like grep. You'll have to get each task log as a separate command. However, this is at least scriptable.

The Hadoop community is working on a more robust log analysis tool that will not only store the job logs on HDFS, but will also give you the ability to perform search and other analyses on these logs. However, this is tool is still a ways out.

Upvotes: 2

WestCoastProjects
WestCoastProjects

Reputation: 63062

This is how we did it (large internet company): we made sure that only v critical messages were logged: but for those messages we actually did use System.err.println. Please keep the aggregate messages per tracker/reducer to only a few KB.

The majority of messages should still use the standard log4j mechanism (which goes to the System logs area)

Upvotes: 0

Related Questions