CODEWITHSUNDEEP
javahadoophadoop2
RBB
RBB

Reputation: 864

Unable to run map reduce job in hadoop 2.7 - Type Mismatch

While running the program getting Error: java.io.IOException: Type mismatch in key from map: expected org.apache.hadoop.io.Text, received org.apache.hadoop.io.LongWritable

I tried more suggestions from google / stack sites. But no luck. Still getting same exception. Any idea, where / what I have missed?

My Imports

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path; 
import org.apache.hadoop.io.IntWritable; 
import org.apache.hadoop.io.LongWritable; 
import org.apache.hadoop.io.Text; 
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.input.TextInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;

My Map class

public static class Map extends Mapper<LongWritable, Text, Text, IntWritable> 
{
    Text k = new Text();


    public void map(Text key, Iterable<IntWritable> value, Context context) 
                throws IOException, InterruptedException {
        String line = value.toString(); 
        StringTokenizer tokenizer = new StringTokenizer(line," "); 
        while (tokenizer.hasMoreTokens()) { 
            String year= tokenizer.nextToken();
            k.set(year);
            String temp= tokenizer.nextToken().trim();
            int v = Integer.parseInt(temp); 
            context.write(k,new IntWritable(v)); 
        }
    }
}

And my reduce class

public static class Reduce extends Reducer<Text, IntWritable, Text, IntWritable>
{

    public void reduce (Text key, Iterable<IntWritable> values, Context context)
            throws IOException, InterruptedException {
        int maxtemp=0;
        for(IntWritable it : values) {
            int temperature= it.get();
            if(maxtemp<temperature)
            {
                maxtemp =temperature;
            }
        }
        context.write(key, new IntWritable(maxtemp)); 
    }
}

And main 

Configuration conf = new Configuration();

Job job = new Job(conf, "MaxTemp");
job.setJarByClass(MaxTemp.class);
job.setMapperClass(Mapper.class);
job.setReducerClass(Reducer.class);

job.setMapOutputKeyClass(Text.class);
job.setMapOutputValueClass(IntWritable.class);

job.setInputFormatClass(TextInputFormat.class);
job.setOutputFormatClass(TextOutputFormat.class);

Path outputPath = new Path(args[1]);

FileInputFormat.addInputPath(job, new Path(args[0]));
FileOutputFormat.setOutputPath(job, new Path(args[1]));

outputPath.getFileSystem(conf).delete(outputPath);

System.exit(job.waitForCompletion(true) ? 0 : 1);

(I have compiled this code with Java 7 in Eclipse IDE (Mars) - exported as runnable jar and Hadoop version is 2.7.0)

Upvotes: 1

Views: 350

Answers (2)

Binary Nerd
Binary Nerd

Reputation: 13927

If you add an @Override annotation to your map function you'll find it doesn't override the map method in Mapper.

If you look at the Javadoc for Mapper (link here) you'll find the map method should look like:

map(KEYIN key, VALUEIN value, org.apache.hadoop.mapreduce.Mapper.Context context)

where as yours looks like

map(Text key, Iterable<IntWritable> value, Context context)

So yours should be:

map(LongWritable key, Text value, Context context)

So because you aren't actually overriding the base map class in Mapper, your method isnt being called an its using the one in Mapper which looks like:

protected void map(KEYIN key, VALUEIN value, 
                     Context context) throws IOException, InterruptedException {
    context.write((KEYOUT) key, (VALUEOUT) value);
}

This will be taking in LongWritable and Text and writting them back out (Identity Mapper) which doesnt match the Text and IntWritable you've told it they should be.

In your driver these lines:

job.setMapperClass(Mapper.class);
job.setReducerClass(Reducer.class);

Should be more like:

job.setMapperClass(Map.class);
job.setReducerClass(Reduce.class);

You need to use your implementations not the base classes.

Upvotes: 1

Amit
Amit

Reputation: 1121

Your mapper definition public static class Map extends Mapper<LongWritable, Text, Text, IntWritable> defines Key is LongWritable and value Text.

However your map method public void map(Text key, Iterable<IntWritable> value, Context context) defines Text as key and Iterable<IntWritable> as value.

Hence your map method should be defined as public void map(LongWritable key, Text value, Context context)

Upvotes: 1

Related Questions