CODEWITHSUNDEEP
rustrust-polars
user655321
user655321

Reputation: 1732

Is it possible to load Parquet data directly from memory?

I have a use case where I will be downloading Parquet data directly into memory (not into the filesystem). Is it possible to load these files as (lazy) dataframes from a Vec<u8>? instead of passing in the path?

Upvotes: 1

Views: 818

Answers (1)

ritchie46
ritchie46

Reputation: 14710

Yes, you can:

use polars::prelude::*;
use std::io::Cursor;

fn main() -> Result<()> {
    let mut df = df![
        "a" => [1, 2, 3]
    ]?;

    // write to buffer
    let mut buf = vec![];
    ParquetWriter::new(&mut buf).finish(&mut df)?;

    // read from buffer
    let reader = Cursor::new(&buf);
    let result = ParquetReader::new(reader).finish()?;

    // validate result
    df.frame_equal(&result);
    Ok(())
}

Upvotes: 2

Related Questions