CODEWITHSUNDEEP
scalaapache-sparkhivehive-metastore
minnu
minnu

Reputation: 67

Check the Metastore for the Table availability in Spark

I want to check if the table is available in a specific hive database before using the below select query. How to get the information from the metastore.

sparkSession.sql("select * from db_name.table_name")

Upvotes: 0

Views: 331

Answers (2)

david gupta
david gupta

Reputation: 56

you can run below command before run your operation on the table

 sparkSession.sql("use databaseName");
 val df = sparkSession.sql("show tables like 'tableName'")
 if(df.head(1).isEmpty == false){
//write the code
}

Upvotes: 1

Raghu
Raghu

Reputation: 1712

Try this

#This will give the list of tables in the database as a dataframe.
tables = sparkSession.sql('SHOW TABLES IN '+db_name)
#You can collect them to bring in a list of row items
tables_list = tables.select('tableName').collect() 
# convert them into a python array
tables_list_arr=[]
for table_name in tables:
    tables_list_arr.append(str(table_name['tableName']))
# Then execute your command
if(table_name in tables_list_arr):
    sparkSession.sql("select * from db_name.table_name")

Upvotes: 0

Related Questions