CODEWITHSUNDEEP
scalaapache-sparkapache-spark-sql
bsky
bsky

Reputation: 20242

java.lang.ClassNotFoundException: org.apache.spark.sql.Dataset

When running a Scala file that uses the Spark Dataset type I get the following stack trace:

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/sql/Dataset
    at java.lang.Class.getDeclaredMethods0(Native Method)
    at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
    at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
    at java.lang.Class.getMethod0(Class.java:3018)
    at java.lang.Class.getMethod(Class.java:1784)
    at com.intellij.rt.execution.application.AppMain.main(AppMain.java:125)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.sql.Dataset
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    ... 6 more

I find this strange because I have the following import:

import org.apache.spark.sql._

Also, in my build.sbt I have the following added to libraryDependencies:

  "org.apache.spark" %% "spark-core" % "1.6.2" % "provided",
  "org.apache.spark" %% "spark-sql" % "1.6.2" % "provided",

Upvotes: 18

Views: 33627

Answers (4)

Galuoises
Galuoises

Reputation: 3283

In IntelliJ 2020.3.2 community edition, go to menu run then edit configurations. Finally, in Modify options select 'Include dependencies with "Provided" scope'.

Upvotes: 10

KP87
KP87

Reputation: 91

Select checkbox 'Include dependencies with "Provided" scope' in Run/Debug Configurations.

image of dropdown & checkbox

Upvotes: 9

soaptree
soaptree

Reputation: 431

Your build.sbt file specified that the spark dependencies are provided to the application's classpath, but it wasn't able to locate them. If you're not running on a cluster, then you can try removing the "provided" from your build.sbt, or put the Spark dependencies on your classpath.

Upvotes: 2

Simon
Simon

Reputation: 6363

If you are executing this standalone you can try removing provided from your dependencies. Provided means that you expect the dependencies to already be on the classpath when you run this application. So the Spark dependencies won't be included in your jar if you use provided.

Upvotes: 46

Related Questions