CODEWITHSUNDEEP
hadoopmapreducehbase
user1580096
user1580096

Reputation: 487

Write Reducer output of a Mapreduce job to a single File

I have written a map-reduce job for the data in HBase. It contains multiple mappers and just a single reducer. The Reducer method takes in the data supplied from the mapper and do some analytic on it. After the processing is complete for all the data in HBase I wanted to write the data back to a file in HDFS through the single Reducer. Presently I am able to write the data to HDFS every time I get new one but unable to figure how to write the final conclusion to HDFS only at last.

Upvotes: 1

Views: 4984

Answers (1)

SSaikia_JtheRocker
SSaikia_JtheRocker

Reputation: 5063

So, if you trying to write a final result from a single reducer to HDFS, you can try any one of the approaches below -

  1. Use Hadoop API FileSystem's create() function to write to HDFS from the reducer.
  2. Emit a single key and value from reducer after the final calculation
  3. Override Reducers cleanup() function and do point (1) there.

Details on 3:

http://hadoop.apache.org/docs/current/api/org/apache/hadoop/mapreduce/Reducer.html#cleanup-org.apache.hadoop.mapreduce.Reducer.Context-

Hope this helps.

Upvotes: 2

Related Questions