CODEWITHSUNDEEP
sql-serverscalaapache-spark
ca9163d9
ca9163d9

Reputation: 29159

Spark program cannot open SQL Server context?

In the file main.scala, 

import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._
import org.apache.spark.SparkConf
import java.sql._

object ConnTest extends App {
  val conf = new SparkConf()
  val sc = new SparkContext(conf.setAppName("Test").setMaster("local[*]"))
  val sqlContext = new org.apache.spark.sql.SQLContext(sc)

  sc.stop()
}

However, the sbt run got the following error.

[info] Compiling 1 Scala source to C:\Users\user1\IdeaProjects\sqlServer\target\scala-2.11\classes...
[error] C:\Users\user1\IdeaProjects\sqlServer\src\main\scala\main.scala:9: type SQLContext is not a member of package org.apache.spark.sql
[error]   val sqlContext = new org.apache.spark.sql.SQLContext(sc)
[error]                                             ^
[error] one error found
[error] (compile:compileIncremental) Compilation failed
[error] Total time: 1 s, completed May 11, 2017 6:11:47 PM

Upvotes: 0

Views: 208

Answers (2)

koiralo
koiralo

Reputation: 23109

You should use Spark 2 which has single entry point SparkSession. you can create SQLContext and SparkContext as 

val sparkSession = SparkSession.builder().master("local[*]").getOrCreate()

val sc = sparkSession.sparkContext
val sqlC = sparkSession.sqlContext

Include the dependency for spark core and spark sql

libraryDependencies += "org.apache.spark" % "spark-core_2.11" % "2.1.1"

libraryDependencies += "org.apache.spark" % "spark-sql_2.11" % "2.1.1"

Upvotes: 1

ashburshui
ashburshui

Reputation: 1410

Your class path is correct, please carefully check you maven or sbt configure

In maven, you should add this specific dependency:

<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-core_2.11</artifactId>
    <version>2.1.1</version>
    <scope>compile</scope>
</dependency>

sbt is similar too, if you want to add this dependency at compile time and package an uber jar with it.

But sometime the spark-core jar is provided by the cluster at runtime, if it is in this case, you can adjust the scope value at you convenient, maybe provided.

What must to be said is that sbt or maven is just dependency control tool, not relevant to spark's execution, you should get your jar package upload on the cluster, and use the spark-submit program to run it!

Please see the community document of the spark example, and have a try:

http://spark.apache.org/examples.html

Good Luck!

Upvotes: 0

Related Questions