CODEWITHSUNDEEP
rust
Kevin
Kevin

Reputation: 3368

Writing to statically-sized file with offset

I am trying to write bytes to a file with random access in the context of downloading random chunks of the file and assembling them in-place. The std::io::Write seem to be limited to sequential writing so I am currently using the following implementation which first needs to download all the chunks and then do big write_all once all the chunks are in memory.

use rand::prelude::*;
use rand; // 0.7.3

fn download<W: std::io::Write>(writer: &mut W) -> std::io::Result<()>
{
    // the size (20) is known
    let mut buf = [0u8; 20];
    
    // simulate downloading four 5 byte chunks at random disjoint slices in *buf*
    for n in (0..20).step_by(5).collect::<Vec<usize>>().choose_multiple(&mut thread_rng(), 4) {
        let n = *n as usize;
        let chunk: [u8; 5] = rand::random();
        buf[n..n+5].clone_from_slice(&chunk);
    }
    
    println!("filled buffer: {:?}", buf);
    
    // finally, do (unbuffered) write
    writer.write_all(&buf)?;
    
    Ok(())
}

fn main() -> std::io::Result<()> {

    let mut f = std::fs::File::create("foo.txt")?;
    download(&mut f)?;
    
    Ok(())
}

playground link

With large files I think this solution may consume to much memory and I would rather write each chunk directly into the file without any intermediate buffer. I see there is write_at in the documentation, but is this method unique for unix systems? I am looking for a generic solution.

Intuitively, I would want something like this to work:

use rand::prelude::*;
use rand; // 0.7.3

fn download<W: std::io::Write>(writer: &mut W) -> std::io::Result<()>
{
    // the size (20) is known
    writer.reserve(20)?;

    // simulate downloading four 5 byte chunks at random disjoint slices in *buf*
    for n in (0..20).step_by(5).collect::<Vec<usize>>().choose_multiple(&mut thread_rng(), 4) {
        let n = *n as usize;
        let chunk: [u8; 5] = rand::random();
        writer.write_at(chunk, n)?;
    }

    Ok(())
}

fn main() -> std::io::Result<()> {

    let mut f = std::fs::File::create("foo.txt")?;
    download(&mut f)?;

    Ok(())
}

Is there any specialization for writing to random access file output? Can I use a memory mapped file? Doing a quick search I could only find the positioned-io crate, which has not been updated in 3 years. Is there anything similar to this crate in the rust standard library?

Upvotes: 2

Views: 782

Answers (1)

cdhowie
cdhowie

Reputation: 169318

You're looking for Seek. File implements both Seek and Write so you just need to bound on Seek as well:

use std::io::{Write, Seek, SeekFrom, Result as IoResult};

fn download<W: Write + Seek>(writer: &mut W) -> IoResult<()> { /* ... */ }

Now you can do writer.seek(SeekFrom::Start(*n as u64 * CHUNK_SIZE))?; before writing out the chunk.

Upvotes: 4

Related Questions