CODEWITHSUNDEEP
javahadooptypes
pellucidcoder
pellucidcoder

Reputation: 127

Hadoop Type Conversion

I have the following signatures for my mapper and reducer classes:

MapperPrime extends Mapper<Text, Text, Text, Text>
ReducerPrime extends Reducer<Text, Text, Text, LongWritable>

When I run the code, I get the following exception when execution reaches the reducer:

java.lang.ClassCastException: org.apache.hadoop.io.LongWritable cannot be cast to org.apache.hadoop.io.Text
    at ReducerPrime.reduce(ReducerPrime.java:1)

When I change the first generic type in ReducerPrime from Text to LongWritable, i.e. 

ReducerPrime extends Reducer<LongWritable, Text, Text, LongWritable>

The exception is no longer thrown. What gives? Why/Where would java be seeing a LongWritable?

Doesn't the input Key,Value type in the Reducer exactly match the ouput Key,Value type in the Mapper?

Upvotes: 1

Views: 368

Answers (1)

OneCricketeer
OneCricketeer

Reputation: 191711

Why/Where would java be seeing a LongWritable

For the Mapper, the key for the TextInputFormat is the file offset, which is a long. The Value is the line of text, so a Text writable object.

Doesn't the input Key,Value type in the Reducer exactly match the ouput Key,Value type in the Mapper?

Yes, it should, though since we can't see the code at the moment, hard to say what you've written as output.

Upvotes: 1

Related Questions