CODEWITHSUNDEEP
javahadoopmapreduce
wazza
wazza

Reputation: 800

No of reducers in mapreduce hadoop

I have only one key emitted from mapper to reducer and I have set no of reducers be 10. So one reducer will run on that key and what other remaining 9 reducers will do?

Upvotes: 4

Views: 94

Answers (2)

Raktotpal Bordoloi
Raktotpal Bordoloi

Reputation: 1057

The other 9 reducers will run until their slot time ends.

They will not have any K/V pairs to process and stop quickly.

You can use custom partitioners to distribute the map outputs to all reducers evenly; - atleast for first level; and finalaly combine through one reducer at very last phase - thus reducing computing load in most of reduce phase.

Upvotes: 0

Binary Nerd
Binary Nerd

Reputation: 13937

The other 9 reducers will run through their lifecycle as normal, they just wont have any key/values to process once they run, so they will stop quickly. Thus you will waste resources while they needlessly run.

You'll generally (most output formats do this) also find you end up with a part file for a reduce that ran but didnt write anything. The part file won't contain any actual data just file metadata, for example gzip headers.

Upvotes: 2

Related Questions