CODEWITHSUNDEEP
hadoop
chet
chet

Reputation: 419

Using a custom Object as key emitted by mapper

I have a situation in which mapper emits as key an object of custom type. It has two fields an intWritable ID, and a data array IntArrayWritable. The implementation is as follows. `

import java.io.*;

import org.apache.hadoop.io.*;

public class PairDocIdPerm implements WritableComparable<PairDocIdPerm> {

    public PairDocIdPerm(){
        this.permId = new IntWritable(-1);
        this.SignaturePerm = new IntArrayWritable();
    }


public IntWritable getPermId() {
        return permId;
    }



    public void setPermId(IntWritable permId) {
        this.permId = permId;
    }



    public IntArrayWritable getSignaturePerm() {
        return SignaturePerm;
    }



    public void setSignaturePerm(IntArrayWritable signaturePerm) {
        SignaturePerm = signaturePerm;
    }

    private IntWritable permId;
    private IntArrayWritable SignaturePerm;

    public PairDocIdPerm(IntWritable permId,IntArrayWritable SignaturePerm) {
   this.permId = permId;
   this.SignaturePerm = SignaturePerm;
   }



    @Override
    public void write(DataOutput out) throws IOException {
    permId.write(out);
    SignaturePerm.write(out);
    }

   @Override
    public void readFields(DataInput in) throws IOException {
    permId.readFields(in);
    SignaturePerm.readFields(in);
    }

    @Override
    public int hashCode() { // same permId must go to same reducer. there fore just permId
     return permId.get();//.hashCode(); 
    }

    @Override
    public boolean equals(Object o) {
     if (o instanceof PairDocIdPerm) {
   PairDocIdPerm tp = (PairDocIdPerm) o;
   return permId.equals(tp.permId) && SignaturePerm.equals(tp.SignaturePerm);
     }
     return false;
    }

    @Override
    public String toString() {
     return permId + "\t" +SignaturePerm.toString(); 
    }

    @Override
    public int compareTo(PairDocIdPerm tp) {
     int cmp = permId.compareTo(tp.permId);
     Writable[] ar, other;
     ar = this.SignaturePerm.get();
     other = tp.SignaturePerm.get();

    if (cmp == 0) {
     for(int i=0;i<ar.length;i++){
         if(((IntWritable)ar[i]).get() == ((IntWritable)other[i]).get()){cmp= 0;continue;}
        else if(((IntWritable)ar[i]).get() < ((IntWritable)other[i]).get()){ return -1;}
         else if(((IntWritable)ar[i]).get() > ((IntWritable)other[i]).get()){return 1;}
     }   
     }

     return cmp;
     //return 1;
   }

   }`

I require the keys with same Id to go to the same reducer with their sort order as coded in the compareTo method. However when i use this, my job execution status is always map100% reduce 0%. The reduce never runs to completion. Is there any thing wrong in this implementation? In general what is the likely problem if reducer status is always 0%.

Upvotes: 1

Views: 1587

Answers (1)

Thomas Jungblut
Thomas Jungblut

Reputation: 20969

I think this might be a possible null pointer exception in the read method:

   @Override
    public void readFields(DataInput in) throws IOException {
    permId.readFields(in);
    SignaturePerm.readFields(in);
    }

permId is null in this case. So what you have to do is this:

IntWritable permId = new IntWritable();

Either in the field initializer or before the read.

However, your code is horrible to read.

Upvotes: 1

Related Questions