CODEWITHSUNDEEP
javahadoopmapreduce
Maverick
Maverick

Reputation: 494

How to fix the "Illegal partition" error in hadoop?

I have written a custom partitioner. When I have number of reduce tasks greater than 1, the job is failing. This is the exception which I'm getting:

 java.io.IOException: Illegal partition for weburl_compositeKey@804746b1 (-1)
 at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:930)
 at org.apache.hadoop.mapred.MapTask$OldOutputCollector.collect(MapTask.java:499)

The code which I have written is

public int getPartition(weburl_compositeKey key, Text value, int numPartitions)
{
    return (key.hashCode()) % numPartitions;
}

This the key.hashCode() equals -719988079 and mod of this value is returning -1.

Appreciate your help on this. Thanks.

Upvotes: 8

Views: 6968

Answers (3)

harpun
harpun

Reputation: 4110

The calculated partition number by your custom Partitioner has to be non-negative. Try:

public int getPartition(weburl_compositeKey key, Text value, int numPartitions)
{
    return (key.hashCode() & Integer.MAX_VALUE) % numPartitions;
}

Upvotes: 24

starkadder
starkadder

Reputation: 311

A warning about using:

public int getPartition(weburl_compositeKey key, Text value, int numPartitions)
{
    return Math.abs(key.hashCode()) % numPartitions;
}

If you hit the case where the key.hashCode() is equal to Integer.MIN_VALUE you will still get a negative partition value. It is an oddity of Java, but Math.abs(Integer.MIN_VALUE) returns Integer.MIN_VALUE ( as in -2147483648). You are safer taking the absolute value of the modulus, as in:

public int getPartition(weburl_compositeKey key, Text value, int numPartitions)
{
    return Math.abs(key.hashCode() % numPartitions);
}

Upvotes: 4

Tanveer
Tanveer

Reputation: 900

Or you can use 

public int getPartition(weburl_compositeKey key, Text value, int numPartitions)
{
    return (key.hashCode() & Integer.MAX_VALUE) % numPartitions;
}

Upvotes: 2

Related Questions