Reputation: 23
I am new to Spark(Spark_SQL), I have loaded data into spark_sql I've also created the table and loaded some data into it.
My question is: where will the schema and the table data be stored?
Upvotes: 0
Views: 1041
Reputation: 191844
where table schema is stored
The Hive metastore
Default is a Derby Database as the "metastore". You can open the folder you ran the Spark job, you'll notice a metastore_db
folder
When using Hive, then the Hive metastore is a MySQL, Postgres, Oracle database that was establish when Hive was installed.
where table will be stored
The Hive warehouse directory
HDFS /apps/hive/warehouse/<databasename>/<tablename>.db/
, for example on a Hortonworks environment.
The default SparkSQL will save a Gzipped Parquet file into that directory as a sequentially numbered part-00xyz
file meaning "partition".
Upvotes: 3