CODEWITHSUNDEEP
scalaapache-sparkhiveapache-spark-sql
Bis
Bis

Reputation: 15

How to resolve this erros "org.apache.spark.SparkException: Requested partitioning does not match the tablename table" in spark-shell

While writing data into hive partitioned table, I am getting below error. org.apache.spark.SparkException: Requested partitioning does not match the tablename table:

I have converted my RDD to a DF using case class and then I am trying to write the data into the existing hive partitioned table. But I am getting his error and as per the printed logs "Requested partitions:" is coming as blank. Partition columns are coming as expected in the hive table.

spark-shell error :-

scala> data1.write.format("hive").partitionBy("category", "state").mode("append").saveAsTable("sampleb.sparkhive6")

org.apache.spark.SparkException: Requested partitioning does not match the sparkhive6 table:

Requested partitions:

Table partitions: category,state

Hive table format :-

hive> describe formatted sparkhive6;

OK

col_name data_type comment

txnno int
txndate string
custno int
amount double
product string
city string
spendby string

Partition Information

col_name data_type comment

category string
state string

Upvotes: 1

Views: 9238

Answers (1)

notNull
notNull

Reputation: 31460

Try with insertInto() function instead of saveAsTable().

scala> data1.write.format("hive")
.partitionBy("category", "state")
.mode("append")
.insertInto("sampleb.sparkhive6")

(or)

Register a temp view on top of the dataframe then write with sql statement to insert data into hive table.

scala> data1.createOrReplaceTempView("temp_vw")
scala> spark.sql("insert into sampleb.sparkhive6 partition(category,state) select txnno,txndate,custno,amount,product,city,spendby,category,state from temp_vw")

Upvotes: 2

Related Questions