CODEWITHSUNDEEP
hadoopmapreduce
Tanya
Tanya

Reputation: 23

incompatibility of hadoop versions

I've developed a MapReduce with reduce-side join on Hadoop 0.20.2 version. All java classes compiles without any error. My program works as it has to. But this version of hadoop is on test virtual machine. On my real cluster I have another version of hadoop (hadoop 2.00-cdh4.1.2). When I want to compile java-classes it doesn't work. I get 100 errors like following:

JoinMapper.java:8: package org.apache.hadoop.filecache does not exist
import org.apache.hadoop.filecache.DistributedCache;
                                  ^
JoinMapper.java:9: package org.apache.hadoop.fs does not exist
import org.apache.hadoop.fs.Path;
                           ^
JoinMapper.java:10: package org.apache.hadoop.io does not exist
import org.apache.hadoop.io.Text;
                           ^
JoinMapper.java:11: package org.apache.hadoop.io does not exist
import org.apache.hadoop.io.*;
^
JoinMapper.java:12: package org.apache.hadoop.mapred does not exist
import org.apache.hadoop.mapred.FileInputFormat;
                               ^
JoinMapper.java:13: package org.apache.hadoop.mapred does not exist
import org.apache.hadoop.mapred.FileOutputFormat;
                               ^
JoinMapper.java:14: package org.apache.hadoop.mapred does not exist
import org.apache.hadoop.mapred.InputFormat;
                               ^
JoinMapper.java:15: package org.apache.hadoop.mapred does not exist
import org.apache.hadoop.mapred.JobClient;
                               ^
JoinMapper.java:16: package org.apache.hadoop.mapred does not exist
import org.apache.hadoop.mapred.JobConf;
                               ^
JoinMapper.java:17: package org.apache.hadoop.mapred does not exist
import org.apache.hadoop.mapred.MapReduceBase;
                               ^
JoinMapper.java:18: package org.apache.hadoop.mapred does not exist
import org.apache.hadoop.mapred.Mapper;
                               ^
JoinMapper.java:19: package org.apache.hadoop.mapred does not exist
import org.apache.hadoop.mapred.OutputCollector;
                               ^
JoinMapper.java:20: package org.apache.hadoop.mapred does not exist
import org.apache.hadoop.mapred.Reporter;
                               ^
JoinMapper.java:21: package org.apache.hadoop.mapred does not exist
import org.apache.hadoop.mapred.TextOutputFormat;
                               ^
JoinMapper.java:22: package org.apache.hadoop.mapreduce.lib.input does not exist
import org.apache.hadoop.mapreduce.lib.input.MultipleInputs;
                                            ^
JoinMapper.java:23: package org.apache.hadoop.util does not exist
import org.apache.hadoop.util.Tool;
                             ^
JoinMapper.java:24: package org.apache.hadoop.util does not exist
import org.apache.hadoop.util.ToolRunner;
                             ^
JoinMapper.java:26: cannot find symbol
symbol: class MapReduceBase
public class JoinMapper extends MapReduceBase implements Mapper<LongWritable, Text, TextPair, Text> {
                                ^
JoinMapper.java:26: cannot find symbol
symbol: class Mapper
public class JoinMapper extends MapReduceBase implements Mapper<LongWritable, Text, TextPair, Text> {
                                                         ^
JoinMapper.java:26: cannot find symbol
symbol: class LongWritable
public class JoinMapper extends MapReduceBase implements Mapper<LongWritable, Text, TextPair, Text> {
                                                                ^
JoinMapper.java:26: cannot find symbol
symbol: class Text
public class JoinMapper extends MapReduceBase implements Mapper<LongWritable, Text, TextPair, Text> {
                                                                              ^
TextPair.java:2: package org.apache.hadoop.io does not exist
import org.apache.hadoop.io.*;
^
TextPair.java:4: cannot find symbol
symbol: class WritableComparable
public class TextPair implements WritableComparable<TextPair> {
                                 ^
TextPair.java:4: interface expected here
public class TextPair implements WritableComparable<TextPair> {
                                                   ^
JoinMapper.java:26: cannot find symbol
symbol: class Text
public class JoinMapper extends MapReduceBase implements Mapper<LongWritable, Text, TextPair, Text> {
                                                                                              ^
JoinMapper.java:26: interface expected here
public class JoinMapper extends MapReduceBase implements Mapper<LongWritable, Text, TextPair, Text> {
                                                               ^
NcdcRecordParser.java:2: package org.apache.hadoop.io does not exist
import org.apache.hadoop.io.*;

I understand this version doesn't have packages that I need. But I don't know how to solve my problem. Can anyone help me?

Upvotes: 3

Views: 4386

Answers (3)

Emil Sit
Emil Sit

Reputation: 23542

Apache Hadoop did a major re-naming of its internal packages between 0.20 and 0.23 (which has been also called 2.0.0). This includes renaming org.apache.hadoop.mapred to org.apache.hadoop.mapreduce.

The Cloudera distribution of Hadoop has separate Maven artifacts that you can depend on to build with the old versions. The names of the artifacts are listed on the CDH4 Maven page; you want the artifacts with version 2.0.0-mr1-cdh4.1.2.

See also Is it better to use the mapred or the mapreduce package to create a Hadoop Job? for some details about whether you may want to port to the newer API.

Upvotes: 1

hudvin
hudvin

Reputation: 63

Try param -Dhadoop.profile=23 for maven. Also, see https://issues.apache.org/jira/browse/HBASE-4327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13098596#comment-13098596

Upvotes: 0

Ben McCracken
Ben McCracken

Reputation: 384

As far as I can tell, these java packages should still exist in the most recent version of Hadoop.

I think your problem is that the hadoop jars are not in your classpath. I get a similar error when I try to compile my map reduce project without them.

If you're using javac, try something like this (this uses the path locations on my Ubuntu CDH4 dev machine):

javac -classpath /usr/lib/hadoop/*:/usr/lib/hadoop-0.20-mapreduce/*:/usr/lib/hadoop-hdfs/* JoinMapper.java TextPair.java NcdcRecordParser.java

Upvotes: 2

Related Questions