CODEWITHSUNDEEP
hadoopconfigurationmapreducehadoop-yarncombiners
polerto
polerto

Reputation: 1790

How to disable hadoop combiner?

In wordcount example, the combiner is explicitly set in

job.setCombinerClass(IntSumReducer.class);

I would like to disable the combiner so that the output of mapper is not processed by the combiner. Is there a way to do that using MR config files (i.e. without modifying and recompiling the wordcount code)?

Thanks

Upvotes: 2

Views: 1098

Answers (1)

allstrives
allstrives

Reputation: 634

Suppose this is your command line

hadoop jar your_hadoop_job.jar your_mr_driver \
command_line_arg1 command_line_arg2 command_line_arg3 \
-libjars all_your_dependency_jars

Here following parameters

  • command_line_arg1
  • command_line_arg2
  • command_line_arg3

will be passed on to your main method as arg[0], arg[1] and arg[3] respectively. Assuming arg[0] and arg[1] is used for identifying input and output folder. You can use arg[3] to pass a boolean flag like ('1' or 'true' or 'yes') to understand if you want to use combiner and accordingly set combiner. Example below (default...it won't set combiner class)

if ( "YyesTrue1".contains(arg[3])){
    job.setCombinerClass(IntSumReducer.class);
}

Upvotes: 1

Related Questions