CODEWITHSUNDEEP
javahadoopmapreduce
Abhijeet Pathak
Abhijeet Pathak

Reputation: 1948

Weird behavior in Reducer of a MapReduce job

Following is the code for a Reducer function I'm using in a MapReduce job. It should return the value from the iterator + custom string ("*---") appended to to each value. But instead it is appending custom string twice.

For example if the value is abc then instead of printing

abc***---

It is printing 

abc***---***---

Why is that happening?

The code:

public static class Reduce extends MapReduceBase implements Reducer<Text, Text, Text, Text> {

    public void reduce(Text key, Iterator<Text> values, OutputCollector<Text, Text> output, Reporter reporter) throws IOException {

        while (values.hasNext()) {

            Text t=values.next();
            String s = "***---";

            t.append(s.getBytes(), 0, s.length());

            output.collect(key, t); 

        }

    }

}

Upvotes: 2

Views: 139

Answers (1)

Matt D
Matt D

Reputation: 3095

Are you using your Reducer class also as a Combiner? If so, the operation in the Reducer will be applied twice: once during the Combine phase (after Map, before shuffle/sort), and again during the Reduce phase.

Upvotes: 3

Related Questions