CODEWITHSUNDEEP
scalaapache-sparkhivescalatest
VJoy
VJoy

Reputation: 33

Not able to create a table locally, getting Hive support is required

Getting error even after setting configuration

config("spark.sql.catalogImplementation","hive")

override def beforeAll(): Unit = {
  super[SharedSparkContext].beforeAll()
  SparkSessionProvider._sparkSession = SparkSession.builder()
    .master("local[*]")
    .config("spark.sql.catalogImplementation","hive")
    .getOrCreate()
}

Edited:

This is how am setting up my local db and tables for testing.

val stgDb = "test_stagingDB"
val stgTbl_exp ="test_stagingDB_expected"
val stgTbl_result="test_stg_table_result"

val trgtDb = "test_activeDB"
val trgtTbl_exp ="test_activeDB_expected"
val trgtTbl_result ="test_activeDB_results"

def setUpDb ={
  println("Set up DB started")
  val localPath="file:/C:/Users/vmurthyms/Code-prdb/prdb/com.rxcorp.prdb"
  spark.sql(s"CREATE DATABASE IF NOT EXISTS test_stagingDB LOCATION '$localPath/test_stagingDB.db'")
  spark.sql(s"CREATE DATABASE IF NOT EXISTS test_activeDB LOCATION '$localPath/test_sctiveDB.db'")
  spark.sql(s"CREATE TABLE IF NOT EXISTS $trgtDb.${trgtTbl_exp}_ina (Id String, Name String)")
  println("Set up DB done")
}
setUpDb

While running spark.sql("CREATE TABLE.., ") cmd , am getting below error: Error: Hive support is required to CREATE Hive TABLE (AS SELECT);; 'CreateTable test_activeDB.test_activeDB_expected_ina, org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, Ignore

org.apache.spark.sql.AnalysisException: Hive support is required to CREATE Hive TABLE (AS SELECT);; 'CreateTable test_activeDB.test_activeDB_expected_ina, org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, Ignore

at org.apache.spark.sql.execution.datasources.HiveOnlyCheck$$anonfun$apply$12.apply(rules.scala:392)
at org.apache.spark.sql.execution.datasources.HiveOnlyCheck$$anonfun$apply$12.apply(rules.scala:390)
at org.apache.spark.sql.catalyst.trees.TreeNode.foreach(TreeNode.scala:117)
at org.apache.spark.sql.execution.datasources.HiveOnlyCheck$.apply(rules.scala:390)
at org.apache.spark.sql.execution.datasources.HiveOnlyCheck$.apply(rules.scala:388)
at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$2.apply(CheckAnalysis.scala:349)
at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$2.apply(CheckAnalysis.scala:349)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$class.checkAnalysis(CheckAnalysis.scala:349)
at org.apache.spark.sql.catalyst.analysis.Analyzer.checkAnalysis(Analyzer.scala:92)
at org.apache.spark.sql.catalyst.analysis.Analyzer.executeAndCheck(Analyzer.scala:105)
at org.apache.spark.sql.execution.QueryExecution.analyzed$lzycompute(QueryExecution.scala:57)
at org.apache.spark.sql.execution.QueryExecution.analyzed(QueryExecution.scala:55)
at org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:47)
at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:74)
at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:641)
at com.rxcorp.prdb.exe.SitecoreAPIExtractTest$$anonfun$2.setUpDb$1(SitecoreAPIExtractTest.scala:127)
at com.rxcorp.prdb.exe.SitecoreAPIExtractTest$$anonfun$2.apply$mcV$sp(SitecoreAPIExtractTest.scala:130)

Upvotes: 0

Views: 1292

Answers (1)

kode
kode

Reputation: 384

It seems you are almost there(your error message is also giving you the clue), you need to call enableHiveSupport() when you are creating spark session. Eg.

SparkSession.builder()
         .master("local[*]")
         .config("spark.sql.catalogImplementation","hive")
         .enableHiveSupport()
         .getOrCreate()

And also when using enableHiveSupport(), setting config("spark.sql.catalogImplementation","hive") looks redundant. I think you can safely comment out that part.

Upvotes: 2

Related Questions