CODEWITHSUNDEEP
hadoop
J. Doe
J. Doe

Reputation: 671

Shuffle and sort, less nodes than pairs

I am trying to understand MapReduce so this is a very noob question. I am looking at the picture below. From my understanding, which might very well be wrong, there is four nodes in the mapping phase and then there is 7 nodes in the shuffle phase. Every key:value pair is moved to different node. My question is what happens if there's only 3 nodes in the shuffle phase? If you have four equal sized key:value pair, can you move them arbitrarily to nodes so it doesn't matter that one is twice the size as the rest or do you split one of the pairs and spread it out evenly?

mapreduce

Upvotes: 0

Views: 398

Answers (1)

gudok
gudok

Reputation: 4189

This image doesn't display actual nodes. Instead, each shuffle/reduce rectangle is a single call to reduce() function. There are 7 of them because 7 distinct keys were emitted by mapper stage. These calls are distributed among reduce tasks. You configure number of reduce tasks yourself with job.setNumReduceTasks(5). If you have one reduce task, all calls will happen there. If you have two reduce tasks, some calls will happen in the first reduce task, others -- in another one (as controlled by Partitioner). If you have 1000 reduce tasks, only some reduce tasks will get reduce() calls, others won't process any data at all.

Reduce tasks are started as separate processes on physical cluster nodes. They may start all simultaneously or not (depends on how many resources you have and also on your scheduler).

Upvotes: 1

Related Questions