CODEWITHSUNDEEP
hadoop
Anand
Anand

Reputation: 1

Hadoop: Do not re-schedule a failed reducer

This is how Hadoop currently works: If a reducer fails (throws a NullPointerException for example), Hadoop will reschedule another reducer to do the task of the reducer that failed.

Is it possible to configure Hadoop to not reschedule failed reducers i.e. if any reducer fails, Hadoop merely reports failure and does nothing else.

Of course, the reducers that did not fail will continue to completion.

Upvotes: 0

Views: 482

Answers (2)

Dmytro Molkov
Dmytro Molkov

Reputation: 11

If you set the configuration to not reschedule failed tasks as soon as the first one fails your jobtracker will fail the job and kill currently running tasks. So what you want to do is pretty much impossible.

Upvotes: 0

Nikolaus Gradwohl
Nikolaus Gradwohl

Reputation: 20124

you can set the mapred.reduce.max.attempts property using the Configuration class the job.xml

setting it to 0 should solve your problem

Upvotes: 1

Related Questions