Kehan Chen
Reputation: 11
How to read a 30G parquet file by python
I am trying to read data from a large parquet file of 30G. My memory do not support default reading with fastparquet in python, so I do not know what I should do to lower the memory usage of the reading process.
Upvotes: 1
Views: 1093
Answers (1)
Micah Kornfield
Reputation: 1698
You can use pyarrow's iter_batches to read back chunks of rows incrementally.
Upvotes: 1
Related Questions
- Create Parquet files from stream in python in memory-efficient manner
- Converting a very very large csv to parquet
- Save a CSV file that's too big to fit into memory into a parquet file
- Can a parquet file exceed 2.1GB?
- Read / Write Parquet files without reading into memory (using Python)
- How to open huge parquet file using Pandas without enough RAM
- Pytorch Dataloader for reading a large parquet/csv file
- Unable to read a parquet file
- Python - read parquet file without pandas
- Reading large number of parquet files: read_parquet vs from_delayed