CODEWITHSUNDEEP
apache-sparkapache-spark-sql
qwertz1123
qwertz1123

Reputation: 1173

Saving empty data frame to parquet

I have a problem saving empty table as parquet. In this case the schema is not preserved and the table cannot be read afterwards. 

myDF.write.format("parquet").partitionBy("part_id").save("aa")

My SQL queries are relying on the same set of tables, and in a case of a table that is empty and therefor cannot be read the queries do not work. Is it any way to save the empty table with metadata ?

With best regards, Michael

Upvotes: 1

Views: 4355

Answers (1)

qwertz1123
qwertz1123

Reputation: 1173

I have removed partitionBy call and after this the metadata was saved correctly

Upvotes: 3

Related Questions