CODEWITHSUNDEEP
scalaapache-spark
wili
wili

Reputation: 123

Spark ERROR：ClassNotFoundException: scala.Cloneable

The IDE what I used is IEAD latest version. The version of sbt is 1.2.8 and the version of Scala is 2.13.0. there is my pom.xml:

<dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-client</artifactId>
            <version>${hadoop.version}</version>
        </dependency>

<dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_2.12</artifactId>
            <version>2.4.3</version>
            <exclusions>
                <exclusion>
                    <groupId>org.slf4j</groupId>
                    <artifactId>slf4j-log4j12</artifactId>
                </exclusion>
            </exclusions>
        </dependency>

I am trying to add those dependencies into my application and change those versions of denpencies. But the error still exists.

<dependency>
            <groupId>org.scala-lang</groupId>
            <artifactId>scala-library</artifactId>
            <version>2.13.0</version>
        </dependency>

        <dependency>
            <groupId>com.typesafe.akka</groupId>
            <artifactId>akka-actor_2.11</artifactId>
            <version>2.3.12</version>
        </dependency>

I had run the code below:

object HelloWorld {

  def connect(): SparkContext = {
    def appName: String = "appName"

//    def master: String = "spark://10.21.49.2:7077"
    def master = "local"
    System.setProperty("hadoop.home.dir", "G:\\JavaWeb\\hadoop-2.7.1")
    val conf = new SparkConf().setAppName(appName).setMaster(master)
    new SparkContext(conf)
  }

  def main(args: Array[String]): Unit = {
    def sc : SparkContext = connect()

    sc.parallelize(Array(2,3))

  }
}

But I get those Exception:

Exception in thread "main" java.lang.NoClassDefFoundError: scala/Cloneable
    at java.lang.ClassLoader.defineClass1(Native Method)
    at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
    at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
    at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
    at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    at HelloWorld$.connect(HelloWorld.scala:11)
    at HelloWorld$.sc$1(HelloWorld.scala:16)
    at HelloWorld$.main(HelloWorld.scala:18)
    at HelloWorld.main(HelloWorld.scala)
Caused by: java.lang.ClassNotFoundException: scala.Cloneable
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    ... 16 more

Thanks, I downgrade the version of the dependency of scala-library to 2.12.x. And there is the new Exception below.

Exception in thread "main" java.lang.NoSuchMethodError: scala.util.matching.Regex.<init>(Ljava/lang/String;Lscala/collection/Seq;)V
    at scala.collection.immutable.StringLike.r(StringLike.scala:281)
    at scala.collection.immutable.StringLike.r$(StringLike.scala:281)
    at scala.collection.immutable.StringOps.r(StringOps.scala:29)
    at scala.collection.immutable.StringLike.r(StringLike.scala:270)
    at scala.collection.immutable.StringLike.r$(StringLike.scala:270)
    at scala.collection.immutable.StringOps.r(StringOps.scala:29)
    at org.apache.spark.util.Utils$.<init>(Utils.scala:1427)
    at org.apache.spark.util.Utils$.<clinit>(Utils.scala)
    at org.apache.spark.SparkConf.loadFromSystemProperties(SparkConf.scala:76)
    at org.apache.spark.SparkConf.<init>(SparkConf.scala:71)
    at org.apache.spark.SparkConf.<init>(SparkConf.scala:58)
    at HelloWorld$.connect(HelloWorld.scala:11)
    at HelloWorld$.sc$1(HelloWorld.scala:16)
    at HelloWorld$.main(HelloWorld.scala:17)
    at HelloWorld.main(HelloWorld.scala)

Upvotes: 2

Views: 3737

Answers (2)

Ram Ghadiyaram
Ram Ghadiyaram

Reputation: 29205

clear indication of version mismatches of spark scala.
docs says...

Spark runs on Java 8+, Python 2.7+/3.4+ and R 3.1+. For the Scala API, Spark 2.4.3 uses Scala 2.12. You will need to use a compatible Scala version (2.12.x).

So, need to downgrade the version of the below dependency

<dependency>
            <groupId>org.scala-lang</groupId>
            <artifactId>scala-library</artifactId>
            <version>2.13.0</version>
        </dependency>

Upvotes: 0

Lamanus
Lamanus

Reputation: 13581

Ok, I found this

Spark runs on Java 8+, Python 2.7+/3.4+ and R 3.1+. For the Scala API, Spark 2.4.3 uses Scala 2.12. You will need to use a compatible Scala version (2.12.x).

You should use scala 2.12, not 2.13 which is released recently and so spark doesn't support yet.

Upvotes: 4

Related Questions