CODEWITHSUNDEEP
hadoopmapreducemapperreducers
Vamshi Krishna B
Vamshi Krishna B

Reputation: 3

Get the total mapping and reducing times in hadoop programmatically

I am trying to calculate the individual total times of Mapping, Shuffling and Reducing by all tasks in my MR code.

I need help retrieving that information for each MapReduce Job.

Can someone post any code snippet that does that calculation?

Upvotes: 0

Views: 140

Answers (1)

Venkat
Venkat

Reputation: 1810

You need to use the JobClient API as shown below: There are however some quirks to the API. Try it let me know i will help you out.

    JobClient client = null;

         Configuration configuration = new Configuration();

        configuration.set("mapred.job.tracker", jobTrackerURL);
        client = new JobClient(new JobConf(configuration));

        while (true) {


            List<JobStatus> jobEntries = getTrackerEntries(jobName,
                        client);


                    for (JobStatus jobStatus : jobEntries) {


                        JobID jobId = jobStatus.getJobID();
                        String trackerJobName = client.getJob(jobId)
                                .getJobName();
                        TaskReport[] mapReports = client
                                .getMapTaskReports(jobId);
                        TaskReport[] reduceReports = client
                                .getReduceTaskReports(jobId);
                        client.getJob(jobId).getJobStatus().getStartTime();


                        int jobMapper = mapReports.length;
                        mapNumber = +jobMapper;
                        int jobReducers = reduceReports.length;
                        reduceNumber = +jobReducers;


                    }
                }

Upvotes: 1

Related Questions