CODEWITHSUNDEEP
apache-spark
ravi
ravi

Reputation: 53

How to run Apache spark Java program in standalone

I have written a java program for spark, but I am not able to run it from the command line.

I have followed the steps given in the Quick start guide, but I am getting the following error. Please help me out with this problem.

Here is the error :

hadoopnod@hadoopnod:~/spark-1.2.1/bin$ ./run-example "SimpleApp " --master local /home/hadoopnod/Spark_Java/target/simple-project-1.0.jarjava.lang.ClassNotFoundException: org.apache.spark.examples.SimpleApp
    at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:270)
    at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:342)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties

Upvotes: 3

Views: 4439

Answers (4)

Brijesh Patel
Brijesh Patel

Reputation: 457

Create a JAR file using following command. You can find the SimpleApp.class file in "target/classes" folder. cd to this directory.

jar cfve file.jar SimpleApp.class

Put this JAR file into your project in target directory. This JAR file contains the dependency of your SimpleApp class while submitting your job to Spark.

cd to your spark directory. I am using spark-1.4.0-bin-hadoop2.6. Your cmd looks like this.

spark-1.4.0-bin-hadoop2.6>

Submit your spark program using Spark Submit. If you have structure like Harsha has explained in another answer then provide 

--class org.apache.spark.examples.SimpleApp

else 

--class SimpleApp

Finally submit your spark program.

spark-1.4.0-bin-hadoop2.6>./bin/spark-submit --class SimpleApp --master local[2] /home/hadoopnod/Spark_Java/target/file.jar

Upvotes: 4

onairos
onairos

Reputation: 11

I had the same issue. If you want to use the command provided by the Spark Quickstart, be sure your project has the same architecture:

find . ./pom.xml ./src ./src/main ./src/main/java ./src/main/java/SimpleApp.java

It may be not the case for you but my pom.xml built my architecture like

./src/main/java/myGroupId/myArtifactId/SimpleApp.java

I moved my class in default package and it worked fine after.

Upvotes: 0

Harsha
Harsha

Reputation: 503

ClassNotFoundException: org.apache.spark.examples.SimpleApp

From the above error, it is clear the reason that it cannot find the class you are trying to execute. Have you bundled your java project into a jar file. If you have any other dependencies while creating your jar file, you need to include them as well.

Assume if you have a project structure like this

simpleapp
 - src/main/java
   - org.apache.spark.examples
     -SimpleApp.java
 - lib
    - dependent.jars (you can put all dependent jars inside lib directory)
 - target
    - simpleapp.jar (after compiling your source)

You can use any build tool or any IDE to bundle your source into a Jar file. After that if you have added spark/bin directory into your path. you can execute below command from your project directory. you need to add --jars $(echo lib/*.jar | tr ' ' ',' ) only if you have dependent libraries in your SimpleApp.java

spark-submit --jars $(echo lib/*.jar | tr ' ' ',' ) --class org.apache.spark.examples.SimpleApp --master local[2]  target/simpleapp.jar

Upvotes: 0

Brad
Brad

Reputation: 15909

The script ./run-example.sh is used to execute the examples included in the distribution. To run the example "SparkPi" do this...

> cd /apps/spark-1.2.0
> ./bin/run-example SparkPi

If you look at how this script executes its just a new user friendly wrapper which actually calls spark-submit.

Here's an example that executes the same "SparkPi" example from above, but using spark-submit

> .bin/spark-submit --class org.apache.spark.examples.SparkPi --master local examples/target/spark-examples_2.10-1.2.0.jar

You should use spark-submit to run your own code.

Upvotes: 0

Related Questions