CODEWITHSUNDEEP
scalaapache-sparksbt
vkubicki
vkubicki

Reputation: 1124

Spark Scala, empty jar in self-contained application

I try to follow the tutorial here: http://spark.apache.org/docs/latest/quick-start.html#self-contained-applications. Using the command sbt package, I obtain an empty jar. The subsequent command:

spark-submit --class SimpleApp --master local[4] target/scala-2.11/simple-project_2.11-1.0.jar

results in:

java.lang.ClassNotFoundException: SimpleApp
        at java.net.URLClassLoader.findClass(Unknown Source)
        at java.lang.ClassLoader.loadClass(Unknown Source)
        at java.lang.ClassLoader.loadClass(Unknown Source)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Unknown Source)
        at org.apache.spark.util.Utils$.classForName(Utils.scala:225)
        at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:693)
        at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:185)
        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:210)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:124)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

I used the correct layout as described int the tutorial, with simple.sbt at the root directory, and the source code in src/scala/SimpleApp.scala.

My configuration is Windows 10, with Scala, Hadoop, Spark and sbt binaries obtained from their respective website.

Upvotes: 1

Views: 484

Answers (1)

vkubicki
vkubicki

Reputation: 1124

sbt enforces a strict folder structure, and I forgot the main intermediate directory. Previously I just used Eclipse which recursively scanned all folders in src.

Adding an intermediate main folder in the directory structure allows compilation and run to proceed as expected:

src
  main
    scala
      SimpleApp.scala

I figured this out with this document http://www.scala-sbt.org/0.13/docs/Directories.html

Upvotes: 1

Related Questions