CODEWITHSUNDEEP
javajarhadoop
Arihant
Arihant

Reputation: 189

How to run a jar file in hadoop?

I have created a jar file using the java file from this blog using following statements

javac -classpath /usr/local/hadoop/hadoop-core-1.0.3.jar -d /home/hduser/dir Dictionary.java

/usr/lib/jvm/jdk1.7.0_07/bin/jar cf Dictionary.jar /home/hduser/dir

Now i have tried running this jar in hadoop by hit and trial of various commands

1hduser@ubuntu:~$ /usr/local/hadoop/bin/hadoop jar Dictionary.jar

Output:

Warning: $HADOOP_HOME is deprecated.

RunJar jarFile [mainClass] args...

2.hduser@ubuntu:~$ /usr/local/hadoop/bin/hadoop jar Dictionary.jar Dictionary

Output:

Warning: $HADOOP_HOME is deprecated.

Exception in thread "main" java.lang.ClassNotFoundException: Dictionary
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:423)
at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at org.apache.hadoop.util.RunJar.main(RunJar.java:149)

How can i run the jar in hadoop? I have the right DFS Locations as per needed by my program.

Upvotes: 12

Views: 78206

Answers (3)

Basapuram Kumar
Basapuram Kumar

Reputation: 179

Use the below command to run the hadoop jar file from the CLI.

hadoop jar <jarFileName> <mainClassname> <AnyCommandLineArguements>

Upvotes: 10

Anirudh
Anirudh

Reputation: 160

I also ran into the same problem and the console does not show much information but just

RunJar jarFile [mainClass] args...

Please check the jar for the package folders location, as a straightforward approach please try is the package starts with com.company...

The "com" folder should be the first level folder when the jar is unpacked

Upvotes: 1

Abhishek Jain
Abhishek Jain

Reputation: 4518

I was able to reproduce your problem. The problem is where you are creating the jar.

Basically, the directory that you are packaging into the jar is confusing the jar file in locating the main class file. Instead if you try doing :

/usr/lib/jvm/jdk1.7.0_07/bin/jar cf Dictionary.jar /home/hduser/dir/Dictionary.class

i.e. package the class file specifically into the jar and then run:

/usr/local/hadoop/bin/hadoop jar Dictionary.jar Dictionary

It just works fine provided that you have a main function in your class called Dictionary.

The problem is when you package a full directory inside a jar then the jar also needs to be aware of the directory structure to locate the class file. For this, we need to have a well defined package hierarchy to define the class location. So, when you are packaging /home/hduser/dir/ into the jar, the jar is not aware of the location of the class file which is located deep inside this directory structure. For this you need to add a package name to your .java file according to the directory structure , for example home.hduser.dir and while running the hadoop jar command specify the class name with the package structure, for example home.hduser.dir.Dictionary.

Upvotes: 20

Related Questions