CODEWITHSUNDEEP
hadoopmapreducehadoop2
Sean Nguyen
Sean Nguyen

Reputation: 13128

Where is the mapper and reducer runtime log in hdfs?

When my mapreduce job finished, i can go to the job history url and see individual reducer/mapper log in there. However, I have a lot of mappers and reducers and I need to down load them all to my local drive to analyze it. I don't know the location of those log files in hdfs. Do you know where it is?

Upvotes: 1

Views: 1248

Answers (3)

Rachit Ahuja
Rachit Ahuja

Reputation: 369

The logs can be found at localhost:50070, under the utilities options

Upvotes: 0

Kumar
Kumar

Reputation: 3990

Actually the userlogs are stored in the local machine only where the nodemanager service runs and where the property yarn.nodemanager.log-dirs set to.

These logs will not be save in the HDFS location. If you want to save these logs in the HDFS then you have to enable Log Aggregation in YARN

Check the below links for more information

Simplifying user logs
YARN Log Aggregation

Similar questions

Where does Hadoop store the logs of YARN applications?

Upvotes: 1

crazy_barber
crazy_barber

Reputation: 46

I presume what you need is a unix command:

yarn logs -applicationId <applicationId>

Application id is revealed during the MR application startup, e.g.

...
15/07/13 10:52:23 INFO input.FileInputFormat: Total input paths to process : 4
15/07/13 10:52:23 INFO mapreduce.JobSubmitter: number of splits:4
15/07/13 10:52:23 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1424784903733_0762
15/07/13 10:52:24 INFO impl.YarnClientImpl: Submitted application application_1424784903733_0762
... : or you can check it up in the web history web page.

What the command does is it dumps all the logs from MR processing to stout.

Upvotes: 2

Related Questions