CODEWITHSUNDEEP
scalaapache-sparksbt
Vitrion
Vitrion

Reputation: 415

SparkSubmit Exception (NoClassDefFoundError), even though SBT compiles and packages sucessfully

I want to use ShapeLogic Scala combined with Spark. I am using Scala 2.11.8, Spark 2.1.1 and ShapeLogic Scala 0.9.0. I sucessfully imported the classes to manage images with Spark. Also, I sucessfully compiled and packed (by using SBT) the following application in order to spark-submitting it to a cluster.

The following application simply opens an image and write it to a folder:

// imageTest.scala
import org.apache.spark.sql.SparkSession
import org.shapelogic.sc.io.LoadImage
import org.shapelogic.sc.image.BufferImage
import org.shapelogic.sc.io.BufferedImageConverter

object imageTestObj {
    def main(args: Array[String]) { 
        // Create a Scala Spark Session
        val spark = SparkSession.builder().appName("imageTest").master("local").getOrCreate();

        val inPathStr = "/home/vitrion/IdeaProjects/imageTest";
        val outPathStr = "/home/vitrion/IdeaProjects/imageTest/output";
        val empty = new BufferImage[Byte](0, 0, 0, Array());
        var a = Array.fill(3)(empty);
        for (i <- 0 to 3) {
            val imagePath = inPathStr + "IMG_" + "%01d".format(i + 1);
            a(i) = LoadImage.loadBufferImage(inPathStr);
        }
        val sc = spark.sparkContext;
        val imgRDD = sc.parallelize(a);
        imgRDD.map { outBufferImage =>
            val imageOpt = BufferedImageConverter.bufferImage2AwtBufferedImage(outBufferImage)
            imageOpt match {
                case Some(bufferedImage) => {
                    LoadImage.saveAWTBufferedImage(bufferedImage, "png", outPathStr)
                    println("Saved " + outPathStr)
                }
                case None => {
                    println("Could not convert image")
                }
            }
        }
    } 
}

This is my SBT file

name := "imageTest"

version := "0.1"

scalaVersion := "2.11.8"

libraryDependencies ++= Seq(
  "org.apache.spark" % "spark-core_2.11" % "2.1.1" % "provided",
  "org.apache.spark" % "spark-sql_2.11" % "2.1.1" % "provided",
  "org.shapelogicscala" %% "shapelogic" % "0.9.0" % "provided"
)

However, the following error appears. When the package SBT command is executed, it seems like the ShapeLogic Scala dependencies are not included in the application JAR:

[vitrion@mstr scala-2.11]$ pwd
/home/vitrion/IdeaProjects/imageTest/target/scala-2.11
[vitrion@mstr scala-2.11]$ ls
classes  imagetest_2.11-0.1.jar  resolution-cache
[vitrion@mstr scala-2.11]$ spark-submit --class imageTestObj imagetest_2.11-0.1.jar 
Exception in thread "main" java.lang.NoClassDefFoundError: org/shapelogic/sc/image/BufferImage
    at imageTestObj.main(imageTest.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:743)
    at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187)
    at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: org.shapelogic.sc.image.BufferImage
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    ... 10 more

I hope someone can help me to solve it? Thank you very much

Upvotes: 0

Views: 968

Answers (2)

user4398985
user4398985

Reputation:

This error says everything:

Caused by: java.lang.ClassNotFoundException: org.shapelogic.sc.image.BufferImage
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)

Add this missing dependency (ShapeLogic class) ------> org.shapelogic.sc.image.BufferImage, that should resolve the issue. Maven or SBT both should give the same error, if you miss this dependency!!

Since you are working on the cluster mode, you can directly add dependencies using --jars on spark-submit, please follow this post for more details.

These threads might help you: Link1 Link2

Upvotes: 2

dumitru
dumitru

Reputation: 2108

Your dependencies listed in sbt files will not be included by default in your jar submitted to spark, so for sbt you have to use a plugin to build an uber/fat jar that would include shapelogicscala classes. You can use this link on SO, How to build an Uber JAR (Fat JAR) using SBT within IntelliJ IDEA? to see how you can manage this with sbt.

Upvotes: 0

Related Questions