CODEWITHSUNDEEP
javahadoop
Itération 122442
Itération 122442

Reputation: 2972

Hadoop job returns exception "classNotFound"

I am trying to launch a word count map reduce on hadoop. When I ask hadoop to do the job, it returns the following:

Exception in thread "main" java.lang.ClassNotFoundException: sdz.hadoop.wordcount.WordCountDriver
    at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:471)
    at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:588)
    at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
    at java.base/java.lang.Class.forName0(Native Method)
    at java.base/java.lang.Class.forName(Class.java:398)
    at org.apache.hadoop.util.RunJar.run(RunJar.java:232)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:153)

This is the output of:

 hadoop jar wordcount.jar sdz.hadoop.wordcount.WordCountDriver ../source.txt ../results

The jar file looks like this:

$jar tf wordcount.jar 
META-INF/
META-INF/MANIFEST.MF
WordCountReducer.class
WordCountMapper.class
WordCountDriver.class

The jar file has been generated with the following command/output:

$jar -cvf wordcount.jar .
added manifest
adding: WordCountReducer.class(in = 1720) (out= 723)(deflated 57%)
adding: WordCountMapper.class(in = 2347) (out= 936)(deflated 60%)
adding: WordCountDriver.class(in = 2278) (out= 1118)(deflated 50%)

The classes have been generated from the following command:

javac -classpath $HADOOP_CLASSPATH WordCount*.java

The java files contain (I know for the useless imports):

Other questions on this said to add the "job.setJarByClass(WordCountDriver.class);". But I already have it. So what is my mistake?

Upvotes: 2

Views: 87

Answers (1)

Ben Watson
Ben Watson

Reputation: 5541

The job will fail as it tries (and fails) to find your classes within the jar at the location:

sdz/hadoop/wordcount/WordCountDriver.class

The quickest solution is to simply remove the package name from your run command (and any package declarations within your classes themselves):

hadoop jar wordcount.jar WordCountDriver ../source.txt ../results

A more stable solution would be to have your jar properly built by a build tool like Maven or Gradle.

Upvotes: 1

Related Questions