CODEWITHSUNDEEP
hadoophashpartitioning
cherri_zj
cherri_zj

Reputation: 109

Partitioning! how does hadoop make it? Use a hash function? what is the default function?

Partitioning is the process of determining which reducer instance will receive which intermediate keys and values. Each mapper must determine for all of its output (key, value) pairs which reducer will receive them. It is necessary that for any key, regardless of which mapper instance generated it, the destination partition is the same Problem: How does hadoop make it? Use a hash function? what is the default function?

Upvotes: 6

Views: 12784

Answers (1)

tommy_o
tommy_o

Reputation: 3783

The default partitioner in Hadoop is the HashPartitioner which has a method called getPartition. It takes key.hashCode() & Integer.MAX_VALUE and finds the modulus using the number of reduce tasks.

For example, if there are 10 reduce tasks, getPartition will return values 0 through 9 for all keys.

Here is the code:

public class HashPartitioner<K, V> extends Partitioner<K, V> {
    public int getPartition(K key, V value, int numReduceTasks) {
        return (key.hashCode() & Integer.MAX_VALUE) % numReduceTasks;
    }
}

To create a custom partitioner, you would extend Partitioner, create a method getPartition, then set your partitioner in the driver code (job.setPartitionerClass(CustomPartitioner.class);). This is particularly helpful if doing secondary sort operations, for example.

Upvotes: 19

Related Questions