CODEWITHSUNDEEP
javahadoop
user3797438
user3797438

Reputation: 485

Sending mapper output to different reducer

I am new to Hadoop and now I am working with java mapper/reducer codes. While working, I came across a problem that I have to pass the output of mapper class to two different reducer class.If it is possible or not.Also can we send two different outputs from same mapper class...Can any one tell me..

Upvotes: 0

Views: 872

Answers (1)

Jacob Thomas
Jacob Thomas

Reputation: 81

I've been trying to do the same. Based on what I found, we cannot have mapper output send to two reducers. But could perform the task that you wanted to do in two reducers in one by differentiating the tasks in the reducer. The reducer can select the task based on some key criteria. I must warn you I'm new to hadoop so may not be the best answer.

The mapper will generate keys like this +-TASK_XXXX. The reducer will then invoke different methods to process TASK_XXXX

Think it is better to have TASK_NAME at the end to ensure effective partitioning.

As for your second question, I believe you can send multiple output from same mapper class to reducer. This post maybe of interest to you Can Hadoop mapper produce multiple keys in output?

The map method would look like 

    @Override
    protected void map(LongWritable key, Text value, Mapper<LongWritable, Text, Text, Text>.Context context) throws IOException, InterruptedException {
        //do stuff 1
        Text outKey1 = new Text(<Your_Original_Key>+"-TASK1");
        context.write(outKey, task1OutValues);

        //do stuff 2
        Text outKey2 = new Text(<Your_Original_Key>+"-TASK2");
        context.write(outKey, task2OutValues);
    }

and reduce method

    @Override
    protected void reduce(Text inkey, Iterable<Text> values, Reducer<Text, Text, Text, Text>.Context context) throws IOException, InterruptedException {
        String key = inKey.toString();
        if(inKey.matches(".*-TASK1$")) {
            processTask1(values);
        } else if(inKey.matches(".*-TASK2$")) {
            processTask2(values);
        }
    }

Upvotes: 1

Related Questions