CODEWITHSUNDEEP
hadoop
Amir Tuval
Amir Tuval

Reputation: 317

Hadoop 0.20.2 reducer throws ArrayIndexOutOfBoundsException when iterating values

I am fairly new to hadoop, however, I've been reading "Hadoop: The definitive guide", so I think I have an understanding of the basic concepts.

I used Hadoop 0.20.2 to run a fairly simple job, but I get the following exception:

java.lang.ArrayIndexOutOfBoundsException: 4096
        at java.io.ByteArrayInputStream.read(ByteArrayInputStream.java:127)
        at java.io.DataInputStream.readInt(DataInputStream.java:373)
        at com.convertro.mapreduce.WritableHit.readFields(Unknown Source)
        at org.apache.hadoop.io.serializer.WritableSerialization$WritableDeseria
lizer.deserialize(WritableSerialization.java:67)
        at org.apache.hadoop.io.serializer.WritableSerialization$WritableDeseria
lizer.deserialize(WritableSerialization.java:40)
        at org.apache.hadoop.mapreduce.ReduceContext.nextKeyValue(ReduceContext.
java:116)
        at org.apache.hadoop.mapreduce.ReduceContext$ValueIterator.next(ReduceCo
ntext.java:163)
        at com.convertro.mapreduce.HitConvertingIterable$HitConvertingIterator.n
ext(HitConvertingIterable.java:35)
        at com.convertro.mapreduce.HitConvertingIterable$HitConvertingIterator.n
ext(HitConvertingIterable.java:1)
        at com.convertro.naive.NaiveHitReducer.reduce(Unknown Source)
        at com.convertro.mapreduce.HitReducer.reduce(Unknown Source)
        at com.convertro.mapreduce.HitReducer.reduce(Unknown Source)
        at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:176)
        at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:566
)
        at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:408)
        at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:2

This happens during the reading of the WritableHit class (the input value to the reduce phase). Below is the WritableHit class code:

 public class WritableHit implements WritableComparable<WritableHit> {

    private Hit hit;

    public WritableHit() {
        this(null);
    }

    public WritableHit(Hit hit) {
        this.hit = hit;
    }

    @Override
    public void readFields(DataInput input) throws IOException {
        String clientName = input.readUTF();
        String clientSiteId = input.readUTF();
        String eventUniqueId = input.readUTF();
        String eventValue = input.readUTF();
        String pageRequested = input.readUTF();
        String refererUrl = input.readUTF();
        String uniqueHitId = input.readUTF();
        String userAgent = input.readUTF();
        String userIdentifier = input.readUTF();
        String userIp = input.readUTF();
        int timestamp = input.readInt();
        int version = input.readInt();

        hit = new Hit(version, uniqueHitId, clientName, clientSiteId, timestamp, userIdentifier, 
                userIp, pageRequested, refererUrl, userAgent, eventUniqueId, eventValue);
    }

    @Override
    public void write(DataOutput output) throws IOException {
        output.writeUTF(hit.getClientName());
        output.writeUTF(hit.getClientSiteId());
        output.writeUTF(hit.getEventUniqueId());
        output.writeUTF(hit.getEventValue());
        output.writeUTF(hit.getPageRequested());
        output.writeUTF(hit.getRefererUrl());
        output.writeUTF(hit.getUniqueHitId());
        output.writeUTF(hit.getUserAgent());
        output.writeUTF(hit.getUserIdentifier());
        output.writeUTF(hit.getUserIp());
        output.write(hit.getTimestamp());
        output.write(hit.getVersion());
    }

    public Hit getHit() {
        return hit;
    }

    @Override
    public int compareTo(WritableHit o) {
        return hit.getUniqueHitId().compareTo(o.getHit().getUniqueHitId());
    }
    }

Any help would be much appreciated.

Thanks

Upvotes: 1

Views: 586

Answers (1)

Amir Tuval
Amir Tuval

Reputation: 317

I figured it out.

Apparently, when you implement a Writable object, you should use the writeInt method, and not the write method.

Once I did that, it worked like a charm.

Upvotes: 1

Related Questions