Reputation: 92179
I have aMapReduce
program as
public static class MapClass extends MapReduceBase implements Mapper<Text, Text, IntWritable, IntWritable> {
private final static IntWritable uno = new IntWritable(1);
private IntWritable citationCount = new IntWritable();
public void map(Text key, Text value, OutputCollector<IntWritable, IntWritable> output, Reporter reporter) throws IOException {
citationCount.set(Integer.parseInt(value.toString()));
output.collect(citationCount, uno);
}
}
public static class Reduce extends MapReduceBase implements Reducer<IntWritable, IntWritable, IntWritable, IntWritable> {
public void reduce(IntWritable key, Iterator<IntWritable> values, OutputCollector<IntWritable, IntWritable> output, Reporter reporter) throws IOException {
int count = 0;
while (values.hasNext()) {
count += values.next().get();
}
output.collect(key, new IntWritable(count));
}
}
I want to run only map
task, where output should be of form <number, 1>
When I run this from command-line I say
$ hadoop jar Hadoop-programs.jar com/hadoop/patent/CitationHistogram input output -Dmapred.reduce.tasks=0
But this is what I see on command-line output
12/07/30 06:13:14 INFO mapred.JobClient: map 50% reduce 0%
12/07/30 06:13:23 INFO mapred.JobClient: map 58% reduce 0%
12/07/30 06:13:26 INFO mapred.JobClient: map 60% reduce 8%
12/07/30 06:13:29 INFO mapred.JobClient: map 68% reduce 8%
12/07/30 06:13:32 INFO mapred.JobClient: map 76% reduce 8%
12/07/30 06:13:35 INFO mapred.JobClient: map 85% reduce 16%
12/07/30 06:13:38 INFO mapred.JobClient: map 93% reduce 16%
12/07/30 06:13:41 INFO mapred.JobClient: map 98% reduce 16%
12/07/30 06:13:44 INFO mapred.JobClient: map 100% reduce 16%
12/07/30 06:13:55 INFO mapred.JobClient: map 100% reduce 69%
12/07/30 06:13:58 INFO mapred.JobClient: map 100% reduce 78%
12/07/30 06:14:01 INFO mapred.JobClient: map 100% reduce 94%
12/07/30 06:14:08 INFO mapred.JobClient: map 100% reduce 100%
When I see the output of job, I see entries like
1 2
13 2
24 1
29 1
31 2
42 3
6796 7
6799 1
6806 1
6815 1
6824 2
Which means data is getting aggregated
How can I not run reducer at all?
Upvotes: 0
Views: 1276
Reputation: 2724
That will work only if you implment ToolRunner.run method and pass it arguments in your main method.
ToolRunner.run(new Configuration(), new YourClasImplmentingToolRunner(), args);
If you don't want to do try setting
job.setNumReduceTasks(0);
or the other option is to set the value in conf and use that configuration in job.
Configuration conf = new Configuration();
conf.set("mapred.reduce.tasks", "0");
Job job = new Job(conf, "My job Name");
Upvotes: 2
Reputation: 7085
Add a space after the -D option and it should work ;)
hadoop jar Hadoop-programs.jar com/hadoop/patent/CitationHistogram input output -D mapred.reduce.tasks=0
Upvotes: 0