CODEWITHSUNDEEP
javahadoophdfs
Liondancer
Liondancer

Reputation: 16489

Hadoop custom Mapper output format to Reducer

Pretty new to HDFS:

Correct me if I am wrong but to my knowledge:

Mapper<Object, Text, Text, Text>

The last two Text are the return types of the Mapper. Say I wanted to return Text, <Text, IntWriteable>. How could I achieve this? Also, where in the documentation should I look?

For example, after the mapper receives the Object and Text, it does some logic with the lines of data in the input file, I want it to return something like

context.write(Text,[Text, IntWriteable])

Upvotes: 0

Views: 1434

Answers (2)

coderz
coderz

Reputation: 4999

You may need to define your own data type. If comparison operation needed, implement WritableComparable interface, otherwise just implement Writable interface is ok.

Here is a sample using WritableComparable interface:

public class MyDataType implements WritableComparable<MyDataType> {
    private Text name;
    private IntWritable age;

    @Override
    public void readFields(DataInput in) throws IOException {
        name.readFields(in);
        age.readFields(in);
    }

    @Override
    public void write(DataOutput out) throws IOException {
        name.write(out);
        age.write(out);
    }

    @Override
    public int compareTo(MyDataType o) {
        int nameCompare = name.compareTo(o.name);
        if(nameCompare != 0) {
            return nameCompare;
        } else {
            return age.compareTo(o.age);
        }
    }
}

Upvotes: 1

Sravan K Reddy
Sravan K Reddy

Reputation: 1082

you have to implement the custom writable in hadoop.I am pointing some sites,might be helpful

https://halalhassan.wordpress.com/2013/12/15/custom-writables-in-hadoop-the-right-way/ http://www.hadoopmaterial.com/2013/10/custom-hadoop-writable-data-type.html

Upvotes: 1

Related Questions