CODEWITHSUNDEEP
go
Mittenchops
Mittenchops

Reputation: 19734

Reading bytes into Go buffer with a fixed stride size

I'd like to read and process 1024 bytes at a time in my file given by filename. I don't understand how to construct the outer loop correctly, especially to accommodate the final stride in which the buffer will contain fewer than 1024 bytes

What I have tried:

fs, _ := os.Open(filename)
defer fs.Close()
n := 1024 // 1kb
buff := make([]byte, n)
for {
    buff = make([]byte, n) // is this initialized correctly?
    n1, err := fs.Read(buff)
    if err != nil {
        if err == io.EOF {
            break
        }
        fmt.Println(err)
        break
    }
    fmt.Println("read n1 bytes...", n1)
    fmt.Println(buff)
}

I have seen the following resources:

Upvotes: 3

Views: 17031

Answers (1)

peterSO
peterSO

Reputation: 166935

Reading bytes into Go buffer with a fixed stride size

read and process 1024 bytes at a time in my file given by filename.

accommodate the final stride in which the buffer will contain fewer than 1024 bytes.

For number of bytes read guarantees, use ioutil.ReadFull. For efficient stream reads, use bufio.Reader. For efficiency, allocate the read buffer once and reuse it.

For example,

package main

import (
    "bufio"
    "fmt"
    "io"
    "io/ioutil"
    "os"
)

func main() {
    stride := 1024
    filename := testname(stride)

    f, err := os.Open(filename)
    if err != nil {
        fmt.Fprintln(os.Stderr, err)
        return
    }
    defer f.Close()

    r := bufio.NewReader(f)
    buf := make([]byte, 0, stride)
    for {
        n, err := io.ReadFull(r, buf[:cap(buf)])
        buf = buf[:n]
        if err != nil {
            if err == io.EOF {
                break
            }
            if err != io.ErrUnexpectedEOF {
                fmt.Fprintln(os.Stderr, err)
                break
            }
        }

        fmt.Println("read n bytes...", n)
        // process buf
    }
}

func testname(stride int) string {
    f, err := ioutil.TempFile("", "test.stride.")
    if err != nil {
        panic(err)
    }
    defer f.Close()
    _, err = f.Write(make([]byte, 2*stride+stride/2))
    if err != nil {
        panic(err)
    }
    return f.Name()
}

Playground: https://play.golang.org/p/iYOY-z7hkoz

Output:

read n bytes... 1024
read n bytes... 1024
read n bytes... 512

Upvotes: 7

Related Questions