CODEWITHSUNDEEP
scalaapache-sparkapache-spark-sql
Martin Senne
Martin Senne

Reputation: 6059

Is there a possibility to keep column order when reading parquet?

Saving a dataframe with columns (e.g. "a", "b") as parquet and then reading the parquet at later point in time does not deliver the same column order (could be "b", "a" f.e.) as the file was saved with.

Unfortunately, I was not able to figure out, how the order is influenced and how I can control it.

How to keep original column order when reading in parquet?

Upvotes: 4

Views: 7959

Answers (1)

dpeacock
dpeacock

Reputation: 2747

PARQUET-188 suggests that column ordering is not part of the parquet spec, so it's probably not a good idea to rely on the ordering. You could however manage this yourself, e.g. by loading/saving the dataframe columns in lexicographical order, or by storing the column names.

Upvotes: 6

Related Questions