CODEWITHSUNDEEP
sqlscalaapache-sparkhive
Haha
Haha

Reputation: 1019

Unable to find Hive table in Spark2 Scala

I have the following function: 

def getMet(spark: SparkSession): DataFrame = {
  val spark = SparkSession.builder().appName("AB Test").enableHiveSupport().getOrCreate()

  val dfMet = spark.sql(s"""SELECT
  10,
  cod_entrep as ID_ENTITE,
  cod_entrep_assu as ID_ENTITE_GARANTE,
  id_notification,
  objet_metier as REF_EXT_OBJET_METIER,
  case when typ_mvt = "S" then 1 else 0 end as TOP_SUPP,
  case when typ_mvt = "S" then dt_capt else null end as DT_SUPP
  FROM dev_lkr_send.pz_send_notification""")
  }

Which returns

exception caught: org.apache.spark.sql.AnalysisException: Table or view not found: pz_send_notification; line 10 pos 0

The table exists in Impala and executing the same steps on spark2-shell works fine.

Any help?

Upvotes: 0

Views: 121

Answers (1)

Haha
Haha

Reputation: 1019

I just found the solution, I had to indicate the hive-site.xml file in the execution command with --files option.

Upvotes: 2

Related Questions