CODEWITHSUNDEEP
javahadoopmapreduce
user4550050
user4550050

Reputation: 621

Progress of mapreduce job

I'm developing a MapReduce application and I want to know the progress of the job I'm running. I am already familiar with the job.mapprocess and job.reducerprocess methods. The problem that those methods only works when the job is finished.

Is there any method that give you the progress of the job on real time while the job is running and not only when it finishs.

Upvotes: 0

Views: 1064

Answers (2)

Mehrez Marouani
Mehrez Marouani

Reputation: 21

In the new Hadoop API, you can access the progress value from the Context object in the mapper or reducer class in the following way:

public class MyMapper extends Mapper<Writable, Writable, Writable, Writable> {

    @Override
    public void map(Writable key, Writable value, Mapper<Writable, Writable, Writable, Writable>.Context context) throws IOException, InterruptedException {
        context.getProgress();
}

Upvotes: 1

yurgis
yurgis

Reputation: 4067

If you mean programmatic access then you need to use JobClient API:

https://hadoop.apache.org/docs/current/api/org/apache/hadoop/mapred/JobClient.html

https://hadoop.apache.org/docs/current/api/org/apache/hadoop/mapred/RunningJob.html

You can submit your job via JobClient:

JobClient jobClient = new JobClient(jobConf);
RunningJob job = jobClient.submitJob(jobConf);
float mapProgress = job.mapProgress();
float redProgress = job.reduceProgress();

Or can look up existing job:

JobClient jobClient = new JobClient(jobConf);
RunningJob job = jobClient.getJob("your_job_id");
...

Upvotes: 0

Related Questions