CODEWITHSUNDEEP
javaapache-spark
theITguy
theITguy

Reputation: 147

Submit Application to spark cluster

I'm having trouble trying to submit an application (job) to a spark (1.0.0) cluster, I already setup the master and attached a worker to it (or at least that's what the WebUI says) following this guide http://spark.apache.org/docs/latest/spark-standalone.html with a pre-builded version that I downloaded from the site... the thing is that when I try to submit an application to the cluster I get the following error:

Exception in thread "main" java.lang.ClassNotFoundException: org.apache.spark.examples.JavaWordCount.java
   at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
   at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
   at java.security.AccessController.doPrivileged(Native Method)
   at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
   at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
   at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
   at java.lang.Class.forName0(Native Method)
   at java.lang.Class.forName(Class.java:270)
   at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:289)
   at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:55)
   at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

now this is the script I'm using, just as it says here http://spark.apache.org/docs/latest/submitting-applications.html:

./bin/spark-submit --verbose --master spark://roman-XPS-L501X:7077 --class org.apache.spark.examples.JavaWordCount.java JavaWordCount.jar

I exported the jar myself and it's in place, also the app is one of the examples given so I assume that nothing is wrong with it. I appreciate any help and I thank you in advance. Also if somebody could explain to me what is the "driver program" that appears and what is its function in the cluster, as you could imagine I'm having a bit of trouble understanding this.

Upvotes: 2

Views: 6072

Answers (1)

rs_atl
rs_atl

Reputation: 8985

First, the driver program is the calling application (i.e. the one that creates the Spark context and defines the operations to be performed).

Second, you are designating the class argument with ".java", which is not the way a fully-qualified class name should be referenced. Your command should be:

./bin/spark-submit --verbose --master spark://roman-XPS-L501X:7077 --class org.apache.spark.examples.JavaWordCount JavaWordCount.jar

Upvotes: 3

Related Questions