CODEWITHSUNDEEP
javahadoopmapreduce
pedron
pedron

Reputation: 237

Strange error in mapreduce class

this error seems trivial, but it won't go away. I have the following class defined:

import java.io.IOException;
import java.util.Iterator;
import java.util.StringTokenizer;

import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapred.FileInputFormat;
import org.apache.hadoop.mapred.FileOutputFormat;
import org.apache.hadoop.mapred.JobClient;
import org.apache.hadoop.mapred.JobConf;
import org.apache.hadoop.mapred.MapReduceBase;
import org.apache.hadoop.mapred.OutputCollector;
import org.apache.hadoop.mapred.Reducer;
import org.apache.hadoop.mapred.Reporter;
import org.apache.hadoop.mapred.TextInputFormat;
import org.apache.hadoop.mapred.TextOutputFormat;
import org.apache.hadoop.mapreduce.Mapper;

public class Anagram_Mapper extends Mapper<LongWritable, Text, Text, Text> {

in the 'main' function i am trying to use JobConf to launch a simple mapreduce:

public static void main(String args[]){
     JobConf conf = new JobConf(Anagram_Mapper.class);
       conf.setJobName("anagram_mapper");

       conf.setOutputKeyClass(Text.class);
       conf.setOutputValueClass(IntWritable.class);

       conf.setMapperClass(Anagram_Mapper.class);
       conf.setCombinerClass(Reduce.class);
       conf.setReducerClass(Reduce.class);

       conf.setInputFormat(TextInputFormat.class);
       conf.setOutputFormat(TextOutputFormat.class);

       FileInputFormat.setInputPaths(conf, new Path(args[0]));
       FileOutputFormat.setOutputPath(conf, new Path(args[1]));

       try {
        JobClient.runJob(conf);
    } catch (IOException e) {
        // TODO Auto-generated catch block
        e.printStackTrace();
    }
}

Eclipse is throwing an error on this line:

conf.setMapperClass(Anagram_Mapper.class);

the error is: 

The method setMapperClass(Class<? extends Mapper>) in the type JobConf 
is not applicable for the arguments (Class<Anagram_Mapper>)

but, as you can see above, my Anagram_Mapper class extends Mapper, right? so, i don't understand why this error....

EDIT: someone posted here, then retracted their post, but it helped steer me in the right direction. apparently i am using: org.apache.hadoop.mapreduce.Mapper

but JobConf.setMapperClass accepts only: org.apache.hadoop.mapred.Mapper

now i'm a little confused about the difference, they seem to be fundamentally the same, and the API tells me they are both valid in Hadoop 2.2.0, the version i'm using....

Upvotes: 2

Views: 5773

Answers (2)

Sandeep bhalla
Sandeep bhalla

Reputation: 31

Faced same error after writing Driver class, below is the error The method setReducerClass(Class) in the type Job is not applicable for the arguments (Class)

reason of getting this error : after creation of reducer class i have immediately passed the class name in setReducerClass(); without defining the reducer class. The function is expecting a class name that actually extends Reducer, it will throw same error until the passed argument is as per the method expected argument type.

Upvotes: 1

Cl&#233;ment MATHIEU
Cl&#233;ment MATHIEU

Reputation: 3171

Indeed you are mixing the old mapred API with the new mapreduce API.

Basically Hadoop mapreduce supports two incompatibles APIs and you have to decide which one to use. I can be confusing because they share classes with same or similar names. You should carefully look at your import statements.

Both API can achieve almost the same thing. mapred has not been deprecated nor removed to no break legacy applications. mapreduce is the same API with a slightly better design.

If you are starting a new project, I would advise to use the new one. But it is not a big deal. The easy fix is to change your org.apache.hadoop.mapreduce.Mapper import statement.

Upvotes: 7

Related Questions