"Decompressing large Zstandard file in Rust results in 'Frame requires too much memory for decoding' error"

Question

I'm using the zstd-rs crate to decompress a large Zstandard file (~30GB) in Rust. My code opens the file, reads it into a buffer, and then uses the copy_decode function from the zstd::stream module to decompress the data and write it to an output file. Here is my current implementation:

use std::fs::File;
use std::io::{self, BufReader, BufWriter, Write};
use zstd::stream::copy_decode;

const SIZE: usize = 1024 * 1024 * 8;

fn decompress_file(input_file: &str, output_file: &str) -> io::Result<()> {
    let input_file = File::open(input_file)?;
    let output_file = File::create(output_file)?;

    let mut reader = BufReader::with_capacity(SIZE, input_file);
    let mut writer = BufWriter::new(output_file);

    copy_decode(&mut reader, &mut writer)?;

    writer.flush()?;

    Ok(())
}

fn main() {
    let input_file = "R:\\reddit\\RS_2021-03.zst";
    let output_file = "R:/out.data";

    match decompress_file(input_file, output_file) {
        Ok(_) => println!("Decompression successful!"),
        Err(e) => println!("An error occurred: {}", e),
    }
}

However, when I run the program, I receive the following error: An error occurred: Frame requires too much memory for decoding. I've looked into the error and it seems that it might be related to the memory allocation for the decompression process, but I'm unsure of the best way to handle this. I've tried to minimize the memory footprint of the program by using buffered reading and writing with a buffer size of 8MB, but I'm still running into this issue.

Does anyone have suggestions for handling large files with zstd-rs or alternative methods for Zstandard decompression in Rust?"

I initially attempted to use the zstd-safe crate for the decompression, but ran into issues with handling the large input file. I then switched to using the zstd-rs crate with buffered reading and writing in the hopes that it would handle larger files more efficiently. I also tried adjusting the size of the buffer used by the BufReader and BufWriter, testing values from 0.5MB up to 8MB.

I expected that using buffered I/O and zstd-rs's stream-based decoding would be able to handle the large file size, but I am still encountering the 'Frame requires too much memory for decoding' error.