CODEWITHSUNDEEP
javaapihadoopmapreduceversion
minhle_r7
minhle_r7

Reputation: 872

How to chain mapper/reducer in Hadoop 1.0.4?

I was using "new" API of Hadoop 1.0.4 (classes in package org.apache.hadoop.mapreduce). When I wanted to chain mapper/reducer, I found out that ChainMapper, ChainReducer are written for the "old" API (classes in package org.apache.hadoop.mapred). What should I do?

Upvotes: 2

Views: 3527

Answers (2)

Jickson T George
Jickson T George

Reputation: 318

I was also searching for the same. I did get the answer and even though its late I thought sharing this may help someone.

From Hadoop 2.0 onwards you can find ChainMapper and ChainReducer in the package org.apache.hadoop.mapreduce.lib.chain

ChainMapper usage pattern: 
...
Job job = new Job(conf, "MyJob");

Configuration map1Conf = new Configuration(false); 
... ChainMapper.addMapper(job, AMap.class, LongWritable.class, Text.class, Text.class, Text.class, true, map1Conf);

Configuration map2Conf = new Configuration(false); 
... ChainMapper.addMapper(job, BMap.class, Text.class, Text.class, LongWritable.class, Text.class, false, map2Conf);

Configuration map3Conf = new Configuration(false); 
... ChainReducer.setReducer(job, CReducer.class, Text.class, Text.class, LongWritable.class, Text.class, false, map3Conf);
...

job.waitForComplettion(true);
...

Upvotes: 5

shazin
shazin

Reputation: 21883

Please read this post. This shows how to make use of two JobConf to enable Chaining of Map Reduce Jobs instead of using ChainMapper/ChainReducer.

Upvotes: 0

Related Questions