CODEWITHSUNDEEP
scalaintellij-ideaapache-sparksbt
user_s
user_s

Reputation: 1078

How to run Spark scala application inside Intellij

I'm trying to run a simple Spark application using Intellij on Hortonworks sandbox. I've opened a new SBT project, then created a Scala class: 

import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._
import org.apache.spark.SparkConf

object SimpleApp {
  def main(args: Array[String]) {
    val logFile = "/root/temp.txt"
    val conf = new SparkConf().setAppName("Simple Application")
    val sc = new SparkContext(conf)
    val logData = sc.textFile(logFile, 2).cache()
    println(logData .count())
  }
}

This is my the build.sbt:

name := "Simple Project"
version := "1.0"
scalaVersion := "2.10.4"
libraryDependencies += "org.apache.spark" % "spark-core" % "1.3.0" % "provided"

Now right clicking on this class -> run throws exception : 

exception in thread main java.lang.noclassdeffounderror: org/apache/Spark/SparkConf

Obviously I'm doing something wrong, but I can see spark libraries on the dependencies list. Any help? (BTW running this program through SBT Scala console works perfectly)

Upvotes: 3

Views: 8740

Answers (2)

Josh
Josh

Reputation: 151

In your build.sbt file, you need two percent signs:

libraryDependencies += "org.apache.spark" %% "spark-core" % "1.6.2" % "provided"

to specify the Scala version of the artifact spark-core.

To run your class in IntelliJ IDEA, you need to also add the Spark library through "File -> Project Structure". Then, under "Libraries", you can add the necessary Spark libs.

Note that objects should not extend App as per the Spark QuickStart

Note that applications should define a main() method instead of extending scala.App. Subclasses of scala.App may not work correctly.

Edit 1: You can also temporarily remove the provided qualifier while testing.

Upvotes: 0

Beyhan Gul
Beyhan Gul

Reputation: 1259

Run 

object SimpleApp extends App {
  def main(args: Array[String]) {
    val logFile = "/root/temp.txt"
    val conf = new SparkConf().setAppName("Simple Application")
    val sc = new SparkContext(conf)
    val logData = sc.textFile(logFile, 2).cache()
    println(logData .count())
  }
}

Upvotes: 3

Related Questions