CODEWITHSUNDEEP
scalaapache-sparkhive
servatj
servatj

Reputation: 196

How spark-shell or Zepellin notebook set HiveContext to SparkSession?

Does anyone know why I can access to an existing hive table from spark-shell or zepelling notebook doing this 

val df = spark.sql("select * from hive_table")

But when I submit a spark jar with a spark object created this way,

     val spark = SparkSession
    .builder()
    .appName("Yet another spark app")
    .config("spark.sql.shuffle.partitions", 18)
    .config("spark.executor.memory", "2g")
    .config("spark.serializer","org.apache.spark.serializer.KryoSerializer")
    .getOrCreate()

I got this

Table or view not found

What I really want is to learn, understand, what the shell and the notebooks are doing for us in order to provide hive context to the SparkSession.

Upvotes: 0

Views: 133

Answers (1)

OneCricketeer
OneCricketeer

Reputation: 191710

When working with Hive, one must instantiate SparkSession with Hive support

You need to call enableHiveSupport() on the session builder

Upvotes: 1

Related Questions