CODEWITHSUNDEEP
hadoopmapreduce
subhankar
subhankar

Reputation: 95

will keys be unique when number of reducers are less

I wanted to ask if different keys can end up in the same reducer. If i build my reducer based on the assumption that keys are same will that be a valid approach. Suppose the number of reducers per job config is 5 and i have unique keys from map output is 10, it should ideally need 10 reducers, will the same reducer method will now receive 2 different kind of keys. Consider Hash partitioning , meant a scenario with less reducers.

Upvotes: 0

Views: 608

Answers (2)

Yann
Yann

Reputation: 361

In general, you cannot assume that 1 reducer will receive only one key. For instance, if your mappers output N keys where N>1, and if you set exactly 1 reducer, then the unique reducer will receive all keys.

But if you control number of different keys = number of reducers, then you may assume that each reducer will receive always the same key. For instance, if your mappers output exactly N keys, and if you set exactly N reducers, then each reducer will receive only 1 key.

Upvotes: 2

Turbero
Turbero

Reputation: 126

If you define a class which implements WritableComparable you can use it as key and define your own rule for the equals ans hashCode methods, this way you can send different keys to the same reducer according to the rules you want.

Upvotes: 0

Related Questions