CODEWITHSUNDEEP
javahadoopmapreducekey-valuereduce
Divyendra
Divyendra

Reputation: 111

Same key different Reducers (HADOOP)?

is it possible to process values with same key on different reducers ? from all mappers i got data with same key and i want to process it with different reducers ? my confusion is that the book says all values with same key will go to same reducer ...

 mapper1(k1,v1),mapper2(k1,v2),mapper3(k1,v3) and so on...

i don't want all data to same reducer ...it should be like,

 reducer1(k1,v1),reducer2(k1,v2)....

and lets say reducer1 produce sum1 and reducer2 produce sum2 and i want that

 sum=sum2+sum1

how should i do that ?

Upvotes: 6

Views: 5400

Answers (2)

Kartikeya Sinha
Kartikeya Sinha

Reputation: 508

You can write a Custom Partitioner for your case, which overrides the default partitioning functionality of Hadoop MR job.

More details here: http://developer.yahoo.com/hadoop/tutorial/module5.html#partitioning

Upvotes: 2

Joe K
Joe K

Reputation: 18424

Data with the same key will always go to the same reducer. But you can choose whatever key you want, so if you want them to go to different reducers, then just choose different keys.

If you want to do an additional combination based on the output from your reducers, then you must do another MapReduce job, with the output from the first job as the input to the next one. This can get ugly fast, so you may wish to look at Cascading, Pig, or Hive to simplify things.

Upvotes: 5

Related Questions