CODEWITHSUNDEEP
hadoop
Shreemay Panhalkar
Shreemay Panhalkar

Reputation: 196

How to retrieve hadoop job map/reduce input/output count

Is there any way to retrieve and print the number of reduce output records after running a hadoop job? I'm iteratively running a map-reduce and I want to stop when my previous reduce output count is same as current map output count.

Upvotes: 1

Views: 817

Answers (1)

Thomas Jungblut
Thomas Jungblut

Reputation: 20969

Depending on your Hadoop version, the names can be different. But in general you can access your counters from the job object.

job.getCounters().findCounter("org.apache.hadoop.mapred.Task$Counter", 
           "REDUCE_OUTPUT_RECORDS").getValue();

Consult the WebUI to see what counters are defined in your job, change the names accordingly.

Upvotes: 1

Related Questions