CODEWITHSUNDEEP
hadoop
A B
A B

Reputation: 1936

How do I filter all the failed Hadoop jobs?

I am trying to find a command that I can use to list all the failed jobs. "hadoop job -list" lists all the jobs. Is there a way to filter the list by status?

Upvotes: 0

Views: 169

Answers (1)

Thanga
Thanga

Reputation: 8101

In Hadoop, There is no way of getting the full list of failed jobs in a single command. hadoop job -list lists only the jobs that are yet running. Only way to get a job's status and report is using the below command 

bin/hadoop job -history <HDFS-OUTPUT-DIR-PATH>

that will throw the full details which include the fail status and in which task it got failed.

JobName: mapsidejjoin
JobConf: hdfs://localhost:50000/tmp/hadoop-thanga/mapred/staging/thanga/.staging/job_201601251423_0001/job.xml
Submitted At: 25-Jan-2016 14:26:05
Launched At: 25-Jan-2016 14:26:05 (0sec)
Finished At: 25-Jan-2016 14:26:24 (19sec)
Status: SUCCESS

Task Summary
============================
Kind    Total   Successful  Failed  Killed  StartTime   FinishTime

Setup   1        1          0       0   25-Jan-2016 14:26:06    25-Jan-2016 14:26:08 (2sec)
Map     1        1          0       0   25-Jan-2016 14:26:08    25-Jan-2016 14:26:11 (2sec)
Reduce  1        1          0       0   25-Jan-2016 14:26:11    25-Jan-2016 14:26:21 (9sec)
Cleanup 1        1          0       0   25-Jan-2016 14:26:21    25-Jan-2016 14:26:24 (2sec)
============================

Upvotes: 2

Related Questions