CODEWITHSUNDEEP
hadoopmapreducereducerscombiners
emesday
emesday

Reputation: 6186

Hadoop mapper emits a unique key. Can I perform reducer after per map?

My mapper emits 'uniq key' - 'very large value' pair.

My reducer doesn't know the key is unique. Thus, the reducer waits until all the mappers are completed.

I tried to use a combiner, but it is not an easy solution for me, because my reducer is very complicated.

My question is how can I perform the reducer after per map? without using a combiner.

Upvotes: 1

Views: 1273

Answers (3)

Niels Basjes
Niels Basjes

Reputation: 10642

If you know in advance your key is unique then you can move all the code from the reducer step into the map and to all the work there.

Upvotes: 2

yura
yura

Reputation: 14645

If your keys are uniq then the is no need to reduce them. Therefore just copy-paste reducer code to mapper and set reducer number to zero. BTW there are many map reduce jobs which do not require reduce step so it is not something strange.

Upvotes: 3

David Medinets
David Medinets

Reputation: 5618

I don't understand your question. You can simply not specify a combiner in the Job configuration.

Upvotes: 0

Related Questions