CODEWITHSUNDEEP
scalaapache-sparkapache-spark-sql
Adeel Hashmi
Adeel Hashmi

Reputation: 837

Accessing Spark.SQL

I am new to Spark. Following the below example in a book, I found that the command below was giving the error. What would be the best way to run a Spark-SQL command, whilst coding in general in Spark?

scala> // Use SQL to create another DataFrame containing the account
summary records
scala> val acSummary = spark.sql("SELECT accNo, sum(tranAmount) as TransTotal FROM trans GROUP BY accNo")
<console>:37: error: not found: value spark

I tried importing import org.apache.spark.SparkContext or using the sc object, but no luck.

Upvotes: 1

Views: 152

Answers (3)

Phasmid
Phasmid

Reputation: 953

Assuming you're in the spark-shell, then first get a sql context thus:

val sqlContext = new org.apache.spark.sql.SQLContext(sc)

Then you can do:

val acSummary = sqlContext.sql("SELECT accNo, sum(tranAmount) as TransTotal FROM trans GROUP BY accNo")

Upvotes: 1

Justin Pihony
Justin Pihony

Reputation: 67135

What version are you using? It appears you're in the shell and this should work, but only in Spark 2+ - otherwise you have to use sqlContext.sql

Upvotes: 0

triggerNZ
triggerNZ

Reputation: 4771

So the value spark that is available in spark-shell is actually an instance of SparkSession (https://spark.apache.org/docs/2.0.2/api/scala/index.html#org.apache.spark.sql.SparkSession)

val spark = SparkSession.builder().getOrCreate()

will give you one.

Upvotes: 0

Related Questions