CODEWITHSUNDEEP
hadoopmapreduce
Ani Menon
Ani Menon

Reputation: 28247

Run a few lines after map & reduce

I have a MapReduce program(in Java) which finds count of words in a document and stores the output as:

word1 10
word2 20
...

I would like to know how to add a few lines to the end of the final output, (something like a finally block of a try and catch) that is I would like to append few words & their scores to the final output.

So my question here, is there a way to add a piece of code which runs after the execution of the reducer so that I may do something after the whole Map & Reduce completes?

Upvotes: 1

Views: 131

Answers (1)

Makubex
Makubex

Reputation: 419

One Reducer : If you are having one reducer, then you can use the context object in the cleanup to write the rank/score for each word. But to do this, you need to have the data that is already written to the output file(word count). I'd suggest you to add a Map or some other object in reduce function to store the word counts. Use that Map object in clean up to find the rank/score and write the result through context object.

Multiple Reducer : If you have multiple reducer, then you have to do the same only in main/run method. But in this case you'd have to read the output file data and then do the calculation before appending to the file. I'd suggest you to use combiners and use a reducer as suggested above to calculate the rank/score.

Upvotes: 3

Related Questions