CODEWITHSUNDEEP
hadoopmapreducejob-scheduling
user974270
user974270

Reputation: 627

how to submit many mapreduce jobs in one terminal?

I am using mapreduce to run the program, to submit one job: I can use following format ./hadoop jar program.jar arguments

in one terminal, this can submit one job, but what if I want to submit 100 jobs? impossible to
open 100 terminals so I am seeking a batch job submitting methods, thanks!

Upvotes: 1

Views: 1074

Answers (1)

Chris White
Chris White

Reputation: 30089

You call ToolRunner.run(..) in a loop within your main method. Be sure to amend your Tool instance to call Job.submit() rather than Job.waitForCompletion() - ensuring your jobs run more in parallel rather than sequentially (you're still limited to how many jobs you can run in parallel by your cluster size / config):

public class MyDriver extends Configured implements Tool {
  public static void main(String args[]) {
    for (int x = 0; x < 100; x++) {
      ToolRunner.run(new MyDriver(), args);
    }
  }

  public int run(String args) {
    Job job = new Job(getConf());

    // job set up
    // ...

    job.submit();
  }
}

Upvotes: 3

Related Questions