CODEWITHSUNDEEP
apache-sparkgzipparquet
YuGagarin
YuGagarin

Reputation: 351

Is gzipped Parquet file splittable in HDFS for Spark?

I get confusing messages when searching and reading answers on the internet on this subject. Anyone can share their experience? I know for a fact that gzipped csv is not, but maybe file internal structures for Parquet are such that it is totally different case for Parquet vs csv?

Upvotes: 14

Views: 9918

Answers (1)

Uwe L. Korn
Uwe L. Korn

Reputation: 8836

Parquet files with GZIP compression are actually splittable. This is because of the internal layout of Parquet files. These are always splittable, independent of the used compression algorithm.

This fact is mainly due to the design of Parquet files that divided in the following parts:

  1. Each Parquet files consists of several RowGroups, these should be the same size as your HDFS Block Size.
  2. Each RowGroup consists of a ColumnChunk per column. Each ColumnChunk in a RowGroup has the same number of Rows.
  3. ColumnChunks are split into Pages, these are probably in the size of 64KiB to 16MiB. Compression is done on a per-page basis, thus a page is the lowest level of parallelisation a job can work on.

You can find a more detailed explanation here: https://github.com/apache/parquet-format#file-format

Upvotes: 28

Related Questions