Can't execute the basic Hadoop Mapreduce Wordcount example

Question

I am trying to run the WordCount example. But I am facing issues with compiling the program.

I get the error:

error: package org.apache.hadoop.mapred does not exist

after executing:

javac -classpath /usr/local/hadoop/share/hadoop/common/hadoop-common-2.7.3.jar -d wordcount_classes WordCount.java

I set up hadoop using this tutorial. I also looked this up on stackoverflow : question and executed the bin/hadoop classpath command in /usr/local/hadoop. This is the output I obtained:

/usr/local/hadoop/etc/hadoop:/usr/local/hadoop/share/hadoop/common/lib/* :/usr/local/hadoop/share/hadoop/common/* :/usr/local/hadoop/share/hadoop/hdfs:/usr/local/hadoop/share/hadoop/hdfs/lib/* :/usr/local/hadoop/share/hadoop/hdfs/* :/usr/local/hadoop/share/hadoop/yarn/lib/* :/usr/local/hadoop/share/hadoop/yarn/* :/usr/local/hadoop/share/hadoop/mapreduce/lib/* :/usr/local/hadoop/share/hadoop/mapreduce/* :/contrib/capacity-scheduler/*.jar

But I don't know what to make of it or what my next step should be! Please help!

Answer 1

You're trying to compile the source code using one of the many hadoop dependency jars (hadoop-common-x.x.x.jar). The jar that contains the mapred package noted in the error message is the hadoop-mapreduce-client-core jar.

I suggest you use a build tool such as Maven or Gradle to build your source code as it will manage transitive dependencies for you.

Alternatively to proceed with your manual invocation of javac, try something like this (untested):

javac -cp '/usr/local/hadoop/share/hadoop/common/*' \
  -cp '/usr/local/hadoop/share/hadoop/hdfs/lib/*' \
  -cp '/usr/local/hadoop/share/hadoop/hdfs/*' \
  -cp '/usr/local/hadoop/share/hadoop/yarn/lib/*' \
  -cp '/usr/local/hadoop/share/hadoop/yarn/*' \
  -cp '/usr/local/hadoop/share/hadoop/mapreduce/lib/*' \ 
  -cp '/usr/local/hadoop/share/hadoop/mapreduce/*' \
  -d wordcount_classes WordCount.java