CODEWITHSUNDEEP
javahadoopmapreduceexecutable-jaroutputformat
manthosh
manthosh

Reputation: 165

Class not found exception when using custom output format for hadoop Map Reduce

I'm trying to run a map reduce job. I created a runnable jar using eclipse and my input format class and output format class are in the same package as the class which starts the job. All these are included in the jar. I set the input and output format class as follows:

job.setInputFormatClass(fully_qualified_name_of_my_input_class.class);
job.setOutputFormatClass(fully_qualified_name_of_my_output_class.class);

But when i execute the jar using hadoop jar myJar.jar i get class not found exception for my output format class but i don't get it for my input format class. How to rectify this?? Here the stacktrace

java.lang.RuntimeException: java.lang.ClassNotFoundException: my-class-name
    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:867)
    at org.apache.hadoop.mapreduce.JobContext.getOutputFormatClass(JobContext.java:235)
    at org.apache.hadoop.mapred.Task.initialize(Task.java:513)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:353)
    at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:416)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
    at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: java.lang.ClassNotFoundException: my-class-name
    at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:266)
    at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:820)
    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:865)
    ... 8 more

Upvotes: 0

Views: 1720

Answers (2)

Manish Agrawal
Manish Agrawal

Reputation: 794

ClassNotFoundException is thrown because JVM is not able to find the class. This can be due to following :-

  1. Really desire class is missing.
  2. Class is there but due to some error. JVM not able to find it.

In 1st case, add the class

In 2nd Case, delete all the class file in bin folder and clean the project.

Now re-run your code. It will resolve the exception.

Upvotes: -1

Mike Park
Mike Park

Reputation: 10941

Try adding the following when you're setting up your job. 

job.setJarByClass(fully_qualified_name_of_my_input_class.class)

See Hadoop query regarding setJarByClass method of Job class for more info.

Upvotes: 2

Related Questions