CODEWITHSUNDEEP
hiveapache-spark-sql
Ham Yu
Ham Yu

Reputation: 357

message:Hive Schema version 1.2.0 does not match metastore's schema version 2.1.0 Metastore is not upgraded or corrupt

enviroment: spark2.11 hive2.2 hadoop2.8.2

hive shell run successfully! and hava no error or warning. but when run application.sh, start failed

/usr/local/spark/bin/spark-submit \
--class cn.spark.sql.Demo \
--num-executors 3 \
--driver-memory 512m \
--executor-memory 512m \
--executor-cores 3 \
--files /usr/local/hive/conf/hive-site.xml \
--driver-class-path /usr/local/hive/lib/mysql-connector-java.jar \
/usr/local/java/sql/sparkstudyjava.jar \

and the error tips:

Exception in thread "main" java.lang.IllegalArgumentException: Error while 
instantiating 'org.apache.spark.sql.hive.HiveSessionState':
...
Caused by: java.lang.IllegalArgumentException: Error while instantiating 
'org.apache.spark.sql.hive.HiveExternalCatalog':
...
Caused by: java.lang.RuntimeException: java.lang.RuntimeException: Unable to 
instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
...
Caused by: java.lang.RuntimeException: Unable to instantiate 
org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
...
Caused by: MetaException(message:Hive Schema version 1.2.0 does not match 
metastore's schema version 2.1.0 Metastore is not upgraded or corrupt)
...

i try many method to solving this errors, but errors still occurs. how to fix?

Upvotes: 8

Views: 9752

Answers (3)

Rakesh saini
Rakesh saini

Reputation: 11

I got the similar issue due to Spark & Hive Version mis-match.Spark 2.0 points to 1.2.0 Hive version and the default Hive i was using was 0.14.0. So by passing version while starting the pyspark will resolve the issue.

pyspark --master yarn --num-executors 1 --executor-memory 512M --conf spark.sql.hive.metastore.version=0.14.0 --conf spark.sql.hive.metastore.jars=/usr/local/hive/apache-hive-0.14.0-bin/*

Upvotes: 1

Sanjay Singh
Sanjay Singh

Reputation: 1057

Sometimes the hive jar in spark could be different than hive version installed, to handle this scenario, You can Pass the jars and version in conf parameter to spark job submit E.g. --conf spark.sql.hive.metastore.version=2.3.0 --conf spark.sql.hive.metastore.jars=/home/apache-hive-2.3.6-bin/lib/*

Upvotes: 3

KiranM
KiranM

Reputation: 1323

Probably hive is referring to another version of hive (which is configured differently). Execute below command & see if output is different from /usr/local/hive.

$which hive

If both are same hive directories, add below properties in hive-site.xml.

<property>
    <name>hive.metastore.schema.verification</name>
    <value>false</value>
</property>

Upvotes: 13

Related Questions