CODEWITHSUNDEEP
hadoopmapreduce
Harsh
Harsh

Reputation: 265

How to spawn a map task at a specific IP address in Hadoop?

Is it possible to spawn a map task (or a TaskTracker with a map task inside) at a specific IP address (or at some node in a rack with a specific rack-id) in Hadoop? If yes, then thanks in advance..

Upvotes: 0

Views: 140

Answers (1)

Chris White
Chris White

Reputation: 30089

'Out of the box' this isn't supported as one of the ideas is to provide distributed processing and fail-over - forcing a task to run on a specific node negates these aims.

Saying that, there may be some ways you could acheive this (and i've certainly see people try where they have hetrogeneous cluster setups for example)

  • You could perform a check in your mapper / reducer setup, throwing an exception if the task executes on a node you don't want it too. You'll need to up the configuration properties for maximum number of failed tasks before the job fails. Not a pretty solution, but again i've seen it work
  • You may be able to implement your own scheduler (instead of the Capacity or Fair schedulers), but i'm not sure how much control you will have

Upvotes: 1

Related Questions