CODEWITHSUNDEEP
gobuffer
BillSmith124
BillSmith124

Reputation: 31

Read exactly n bytes unless EOF?

I'm using a function that returns an io.Reader to download a file from the Internet.

I want to process the file in exactly 2048 chunks until it's no longer possible because of EOF.

The io.ReadFull function is almost what I want:

buf := make([]byte, 2048)

for {
    if _, err := io.ReadFull(reader, buf); err == io.EOF {
        return io.ErrUnexpectedEOF
    } else if err != nil {
        return err
    }

    // Do processing on buf
}

The problem with this is that not all files are a multiple of 2048 bytes, so the last chunk may only be e.g. 500 bytes, io.ReadFull will therefore return ErrUnexpectedEOF and the last chunk is discarded.

A function name to summarize what I want could be io.ReadFullUnlessLastChunk, so ErrUnexpectedEOF is not returned if the reason that buf cannot be filled with 2048 bytes, is that the file is EOF after e.g. 500 bytes. However, in any other case ErrUnexpectedEOF should be returned as a problem has occured.

What could I do to accomplish this?

Another problem is that reading only 2048 bytes at the time directly from the network seems to have much overhead, if I could get 256 KB from network into a buffer, and then take the 2048 bytes I need from that buffer instead, that would be better.

Upvotes: 2

Views: 2867

Answers (1)

peterSO
peterSO

Reputation: 166925

For example,

package main

import (
    "bufio"
    "fmt"
    "io"
    "os"
)

func readChunks(r io.Reader) error {
    if _, ok := r.(*bufio.Reader); !ok {
        r = bufio.NewReader(r)
    }
    buf := make([]byte, 0, 2048)
    for {
        n, err := io.ReadFull(r, buf[:cap(buf)])
        buf = buf[:n]
        if err != nil {
            if err == io.EOF {
                break
            }
            if err != io.ErrUnexpectedEOF {
                return err
            }
        }

        // Process buf
        fmt.Println(len(buf))

    }
    return nil
}

func main() {
    fName := `test.file`
    f, err := os.Open(fName)
    if err != nil {
        fmt.Println(err)
        return
    }
    defer f.Close()

    err = readChunks(f)
    if err != nil {
        fmt.Println(err)
        return
    }
}

Upvotes: 1

Related Questions