CODEWITHSUNDEEP
apache-sparkapache-spark-sql
Sayantan Sarkar
Sayantan Sarkar

Reputation: 9

Spark org.apache.spark.sql.catalyst.analysis.UnresolvedException error in loading Hive table

While trying to load data from a dataset into Hive table getting the error:

org.apache.spark.sql.catalyst.analysis.UnresolvedException: Invalid call to dataType on unresolved object, tree: 'ipl_appl_signed_date

My dataset contains same columns as the Hive table and the column for which am getting the error has Date datatype in my code(Java) as well as in Hive.

java code:

Date IPL_APPL_SIGNED_DATE   =rs.getDate("DTL.IPL_APPL_SIGNED_DATE"); //using jdbc to get record.

Encoder<DimPolicy> encoder = Encoders.bean(Foo.class);

Dataset<DimPolicy> test=spark.createDataset(allRows,encoder); //spark is the spark session

test.write().mode("append").insertInto("someSchema.someTable"); //

Upvotes: 1

Views: 2521

Answers (1)

Jacek Laskowski
Jacek Laskowski

Reputation: 74619

I think the issue is due to a bug in Spark i.e. [SPARK-26379] Use dummy TimeZoneId for CurrentTimestamp to avoid UnresolvedException in CurrentBatchTimestamp, that got fixed in 2.3.3, 2.4.1, 3.0.0.

A solution is to downgrade to the version of Spark that is unaffected by the bug (or wait for a new version).

Upvotes: 1

Related Questions