CODEWITHSUNDEEP
hadoopmapreduce
HHH
HHH

Reputation: 6485

Submitting a Hadoop job

I need to constantly get the mappers' and reducers' running time. I have submitted the job as follows.

 JobClient jobclient = new JobClient(conf);
 RunningJob runjob = jobclient.submitJob(conf);          


 TaskReport [] maps = jobclient.getMapTaskReports(runjob.getID());

 long mapDuration = 0;
 for(TaskReport rpt: maps){
    mapDuration += rpt.getFinishTime() - rpt.getStartTime();
 }

However when I run the program, it seems like the job is not submitted and the mapper never starts. How can I use JobClient.runJob(conf) and still be able to get the running time?

Upvotes: 2

Views: 1277

Answers (1)

Chris Gerken
Chris Gerken

Reputation: 16390

The submitJob() method returns control immediately to the calling program without waiting for the hadoop Job to start, much less complete. If you want to wait then use the waitForCompletion() method which returns only after the hadoop job has finished. I think you want something in between since you want to run subsequent code after the submit but before the complete.

I suggest you put your follow-on code in a loop that continues until the job is complete (Use the isComplete() method for that test) and observe the mappers and reducers as the job progresses. You probably want to put a Thread.sleep(xxx) in the loop somewhere, too.

To respond to your comment, you want to...

job.waitForCompletion();
TaskCompletionEvent event[] = job.getTaskCompletionEvents();
for (int i = 0; i < event.length(); i++) {
    System.out.println("Task "+i+" took "+event[i].getTaskRunTime()+" ms");
}

Upvotes: 1

Related Questions