CODEWITHSUNDEEP
javaapihadoop
Dinis Gostev
Dinis Gostev

Reputation: 41

How to get specific MR counters from Hadoop java API?

In hadoop ui, we can see standard MR counters in separated columns: map-task, reduce-task and total.

enter image description here

But I don't see such methods in hadoop java API. How can we get separated counters from java api, e.g. how to get CPU time spent (ms) values for Map and Reduce columns?

Upvotes: 2

Views: 125

Answers (1)

Dinis Gostev
Dinis Gostev

Reputation: 41

If you want to get separated task counters, you need to use getTaskReports() method to your hadoop job. After that you can use getTaskCounter() and findCounter() methods to find counter that you need, for each task separately.

For example, how to get your map gc-time-millis counter:

Long result = 0L;    
TaskReport[] taskReports = job.getHadoopJob().getTaskReports(TaskType.MAP);
for (TaskReport taskReport : taskReports) {
   try {
       result += taskReport.getTaskCounters().findCounter(TaskCounter.GC_TIME_MILLIS).getValue();
       } catch (Exception ignored) {}
}

Upvotes: 1

Related Questions