CODEWITHSUNDEEP
javascalaapache-sparkintellij-idea
Mubin
Mubin

Reputation: 4425

Apache Spark unable to find class

I'm trying to develop stand alone app using apache spark alongside scala sbt I'm getting this error again and again.

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/SparkConf at AuctionsApp$.main(AuctionsApp.scala:5)

Here is code snippet.

import org.apache.spark.SparkConf

object AuctionsApp {
  def main(args: Array[String]): Unit = {
    val conf = new SparkConf().setAppName("AuctionsApp")
    println(conf)
  }
}

build.sbt file

name := "AuctionsApp"

version := "1.0"

scalaVersion := "2.10.5"

libraryDependencies += "org.apache.spark" % "spark-core_2.10" % "1.2.0-cdh5.3.2" % "provided"


resolvers += "Cloudera Repository" at "https://repository.cloudera.com/artifactory/cloudera-repos/"

I can navigate to SparkConf file by pressing command and click on class name in import statement, means that it resides there alongside the other code.

Project build was successful too.

any help is appreciated.

PS. using IntelliJ Idea CE version 2016.3

Upvotes: 2

Views: 796

Answers (1)

Yuval Itzchakov
Yuval Itzchakov

Reputation: 149518

Note that in SBT, you specify provided, which means the JAR containing the class is assumed to be present on the classpath when the application starts, which isn't the case when debugging locally.

To get around this issue we create an additional project which is meant solely for debugging Spark locally. First, we declare the dependencies on Spark outside of the projects definition:

lazy val sparkDeps = Seq(
  "org.apache.spark" %% "spark-core" % "1.2.0-cdh5.3.2"
)

Then we create the new project which depends on our main project and map over Sparks dependencies and add the compile annotation onto the module:

lazy val sparkProject = 
  project in file("spark-project")
  .settings(
   // Add your settings and such
  )

lazy val sparkDebugger = project in file("spark-debugger")
  .dependsOn(sparkProject)
  .settings(
    libraryDependencies ++= sparkDeps.map(_ % "compile")
)

And when we want to debug Spark locally, we make our main method load up from the sparkDebugger project in the Debug Configuration window:

Spark Debug Configuration

Upvotes: 3

Related Questions