CODEWITHSUNDEEP
apache-sparkhivepysparkcloudera-cdh
hangkongwang
hangkongwang

Reputation: 215

pyspark dataframe save to hive table can not be found

we have installed our cluster via cdh6.2. use pyspark create a dataFrame, then save it to hive.

the file is created in warehouse correctly, but it can not be found in hive or impala using show tables.

it can be found by spark sql using sql.('show tables'). But it only show the table be created by the spark code before, which means it can not see tables create via hive or impala console.

so I think it maybe the spark code doesn't collect to the Hive Metastore server.

but I don't know how to setup it to the Hive Metastore server.

Upvotes: 0

Views: 328

Answers (1)

user8143344
user8143344

Reputation:

In order to connect to hive metastore you need to copy the hive-site.xml file into spark/conf directory. Try the following:

ln -s /usr/lib/hive/conf/hive-site.xml /usr/lib/spark/conf/hive-site.xml

Upvotes: 0

Related Questions