Spark sql can't find table in hive in HDP

Question

I use HDP3.1 and I added Spark2, Hive and Other services which are needed. I turned of the ACID feature in Hive. The spark job can't find the table in hive. But the table exists in Hive. The exception likes: org.apache.spark.sql.AnalysisException: Table or view not found There is hive-site.xml in Spark's conf folder. It is automaticly created by HDP. But it isn't same as the file in hive's conf folder. And from the log, the spark can get the thrift URI of hive correctly. I use spark sql and created one hive table in spark-shell. I found the table was created in the fold which is specified by spark.sql.warehouse.dir. I changed its value to the value of hive.metastore.warehouse.dir. But the problem is still there. I also enabled hive support when creating spark session.

val ss = SparkSession.builder().appName("统计").enableHiveSupport().getOrCreate()

Answer 1

In HDP 3.0 and later, Spark and Hive use independent catalogs for accessing SparkSQL or Hive tables on the same or different platforms.

Spark, by default, only reads the Spark Catalog. And, this means Spark applications that attempt to read/write to tables created using hive CLI will fail with table not found exception.

Workaround:

• Create the table in Hive CLI and Spark SQL
• Hive Warehouse Connector

Answer 2

You can use hivewarehouse connector and use llap in hive conf

Answer 3

There is metastore.catalog.default in hive-site.xml in spark's conf folder. It value is spark. It should be changed to hive. And btw, we should disable ACID feature of hive.