CODEWITHSUNDEEP
javahadoopdistributed-computing
azec.me
azec.me

Reputation: 5078

Hadoop jobs hanging waiting to be killed

I have multiple Hadoop jobs doing different processing. When exception occurs in some of these (custom business exception) it is propagated to the map() method, job is killed right away.

However some of the jobs are throwing exception in the setup() method in mapper class. When this happens job hangs for 40 minutes (my Hadoop timer has to be set to this), and with task re-run policy, I have to wait 4*40 = 160 minutes. In this particular case IllegalArgumentException is being thrown by one of business classes.

How could I handle this long "job hanging on cloud" period, but keeping my timer as is?

Upvotes: 1

Views: 2307

Answers (1)

Brainlag
Brainlag

Reputation: 628

You can kill a job on the command line with hadoop job -kill <jobid>

Upvotes: 2

Related Questions