CODEWITHSUNDEEP
daskcudf
Vibhu Jawa
Vibhu Jawa

Reputation: 88

How to read a single large parquet file into multiple partitions using dask/dask-cudf?

I am trying to read a single large parquet file (size > gpu_size), using dask_cudf/dask but it is currently reading it into a single partition, which i am guessing is the expected behavior inferring from the doc-string: 

dask.dataframe.read_parquet(path, columns=None, filters=None, categories=None, index=None, storage_options=None, engine='auto', gather_statistics=None, **kwargs):

    Read a Parquet file into a Dask DataFrame
    This reads a directory of Parquet data into a Dask.dataframe, one file per partition. 
    It selects the index among the sorted columns if any exist.

Is there a work-around i can do read it into multiple partitions ?

Upvotes: 3

Views: 2471

Answers (1)

MRocklin
MRocklin

Reputation: 57319

Parquet datasets can be saved into separate files. Each file may contain separate row groups. Dask Dataframe reads each Parquet row group into a separate partition.

Based on what you're saying it sounds like your dataset has only a single row group. If that is the case then unfortunately there is nothing that Dask can really do here.

You might want to go back to the source of the data to see how it was saved and verify that whatever process is saving this dataset does it in a way where it is not creating very large row groups.

Upvotes: 1

Related Questions