CODEWITHSUNDEEP
parquet
jpwkeeper
jpwkeeper

Reputation: 341

Parquet file with uneven columns

I'm trying to figure out how to write a parquet file where the columns do not contain the same number of rows per Row Group. For example, my first column might be a value sampled at 10Hz, while my second column may be a value sampled at only 5Hz. I'd rather not repeat values in the slower column since this can lead to computational errors. However, I cannot write columns of two different sizes to the same Row Group, so how can I accomplish this? I'm attempting to do this with ParquetSharp.

Upvotes: 1

Views: 1051

Answers (1)

namesis
namesis

Reputation: 157

It is not possible for the columns in a parquet file to have different row counts.

It is not explicit in the documentation but if you look on https://parquet.apache.org/documentation/latest/#metadata, you will see that a RowGroup has a num_rows and several ColumnChunks that do not themselves have individual row numbers.

Upvotes: 0

Related Questions