CODEWITHSUNDEEP
javahadoopmapreduce
fanbondi
fanbondi

Reputation: 1047

Change reducer sorting order

I have the following output from my reducer 

    Key            Value
1 1998-01-05         45
10 1998-01-09        20
2 1998-01-06         68
3 1998-01-07         85
4 1998-01-08         85

Lexicographically this is correct but I want it to sorted in Natural order e.g. 

  Key              Value
1 1998-01-05         45
2 1998-01-06         68
3 1998-01-07         85
4 1998-01-08         85
10 1998-01-09        20

I wrote a KeyComparator to achieve this and below is the code but even this did not work out. 

public static class KeyComparator extends WritableComparator {
            protected KeyComparator() {
                    super(IntWritable.class, true);
            }

            @SuppressWarnings("rawtypes")
            public int compare(WritableComparable w1, WritableComparable w2) {
                    IntWritable t1 = (IntWritable) w1;
                    IntWritable t2 = (IntWritable) w2;
                    String t1Items = t1.toString();
                    String t2Items = t2.toString();
                    return t1Items.compareTo(t2Items);
            }
    }

Note my mapper output have the same format as the reducer but reducer is just outputting the maximum value.

What am I missing

Upvotes: 0

Views: 533

Answers (3)

Marko Topolnik
Marko Topolnik

Reputation: 200266

The correct way is the trivial way in this case:

public int compare(WritableComparable w1, WritableComparable w2) {
  return w1.compareTo(w2);
}

IntWritable already implements the Comparable interface the proper way.

That said, you may not even need your custom comparator class.

Upvotes: 1

AlexR
AlexR

Reputation: 115388

You are comparing strings 

               String t1Items = t1.toString();
               String t2Items = t2.toString();
               return t1Items.compareTo(t2Items);

You should not do this. Compare the numbers instead. I have no idea what IntWritable is but you should not create string from it. Extract integers and compare them directly.

Upvotes: 2

Peter Lawrey
Peter Lawrey

Reputation: 533820

You are comparing Strings instead of comparing values. "10" < "2" even though 10 > 2.

You need to get the first field from your IntWritable or parse the first number and compare it.

BTW: If you are going to use toString(), you don't need to cast it first as all Object support this method.

Upvotes: 3

Related Questions