CODEWITHSUNDEEP
hadoopcounterchaining
Reuben Tanner
Reuben Tanner

Reputation: 5565

How to get the mapper output byte counter

I am chaining an undetermined amount of map reduce jobs together for a parallel BFS shortest path algorithm and when the path cannot be determined, my jobs loop infinitely without producing any records. I figured the best way to check this is to get the Map Output Bytes counter that is maintained by hadoop.

How can I get access to this counter?

Upvotes: 1

Views: 874

Answers (1)

Reuben Tanner
Reuben Tanner

Reputation: 5565

To get the map output bytes counter produced by the job, use

long outputBytes = job.getCounters().findCounter("org.apache.hadoop.mapred.Task$Counter", "MAP_OUTPUT_BYTES").getValue();

See http://lintool.github.io/Cloud9/docs/content/counters.html for more counter groups and names

Upvotes: 2

Related Questions