CODEWITHSUNDEEP
hadoopmapreducehadoop2
akhil.cs
akhil.cs

Reputation: 691

Issue when working with ArrayWritables

I'm a beginner in Hadoop and was working with the ArrayWritables in Hadoop map-reduce.

And this is the Mapper code I'm using :-

public class Base_Mapper extends Mapper<LongWritable, Text, Text, IntWritable> {
    String currLine[] = new String[1000];
    Text K = new Text();

    public void map(LongWritable key, Text value, Context context)
            throws IOException, InterruptedException {
        String line = value.toString();
        currLine = line.split("");
        int count = 0;
        for (int i = 0; i < currLine.length; i++) {
            String currToken = currLine[i];
            count++;
            K.set(currToken);
            context.write(K, new IntWritable(count));
        }

    }
}

Reducer :-

public class Base_Reducer extends Reducer<Text, IntWritable,Text, IntArrayWritable> {

    public void reduce(Text key, Iterable<IntWritable> values, Context context)
            throws IOException, InterruptedException {
        IntArrayWritable finalArray = new IntArrayWritable();
        IntWritable[] arr = new IntWritable[1000];
        for (int i = 0; i < 150; i++)
            arr[i] = new IntWritable(0);
        int redCount = 0;
        for (IntWritable val : values) {
            int thisValue = val.get();
            for (int i = 1; i <= 150; i++) {
                if (thisValue == i)
                    arr[i - 1] = new IntWritable(redCount++);
            }
        }
        finalArray.set(arr);
        context.write(key, finalArray);
    }

}

I'm using IntArrayWritable as subclass of ArrayWritable as shown below :-

import org.apache.hadoop.io.ArrayWritable;
import org.apache.hadoop.io.IntWritable;

public class IntArrayWritable extends ArrayWritable {
    public IntArrayWritable() {
        super(IntWritable.class);
    }

    public IntArrayWritable(IntWritable[] values) {
        super(IntWritable.class, values);
    }
}

My Intended output of the Job was some set of Bases as key(which is correct) and an array of IntWritables as value. But I'm getting the output as:-

    com.feathersoft.Base.IntArrayWritable@30374534
A   com.feathersoft.Base.IntArrayWritable@7ca071a6
C   com.feathersoft.Base.IntArrayWritable@9858936
G   com.feathersoft.Base.IntArrayWritable@1df33d1c
N   com.feathersoft.Base.IntArrayWritable@4c3108a0
T   com.feathersoft.Base.IntArrayWritable@272d6774

What are all changes I have to make inorder to resolve this issue ?

Upvotes: 2

Views: 975

Answers (1)

vanekjar
vanekjar

Reputation: 2406

You need to override default behavior of toString() method in your IntArrayWritable implementation.

Please try this:

import org.apache.hadoop.io.ArrayWritable;
import org.apache.hadoop.io.IntWritable;

public class IntArrayWritable extends ArrayWritable {
    public IntArrayWritable() {
        super(IntWritable.class);
    }

    public IntArrayWritable(IntWritable[] values) {
        super(IntWritable.class, values);
    }

    @Override
    public String toString() {
        StringBuilder sb = new StringBuilder("[");

        for (String s : super.toStrings())
        {
            sb.append(s).append(" ");
        }

        sb.append("]")
        return sb.toString();
    }
}

If you liked this answer please mark it as accepted. Thank you.

Upvotes: 3

Related Questions