Ravi Reddy
Ravi Reddy

Reputation: 23

Spark_SQL where will the schema and the table data be stored?

I am new to Spark(Spark_SQL), I have loaded data into spark_sql I've also created the table and loaded some data into it.
My question is: where will the schema and the table data be stored?

Upvotes: 0

Views: 1041

Answers (1)

OneCricketeer
OneCricketeer

Reputation: 191844

where table schema is stored

The Hive metastore

Default is a Derby Database as the "metastore". You can open the folder you ran the Spark job, you'll notice a metastore_db folder

When using Hive, then the Hive metastore is a MySQL, Postgres, Oracle database that was establish when Hive was installed.

where table will be stored

The Hive warehouse directory

HDFS /apps/hive/warehouse/<databasename>/<tablename>.db/, for example on a Hortonworks environment.

The default SparkSQL will save a Gzipped Parquet file into that directory as a sequentially numbered part-00xyz file meaning "partition".

Upvotes: 3

Related Questions