CODEWITHSUNDEEP
go
stian
stian

Reputation: 1987

limitation on bytes.Buffer?

I am trying to gzip a slice of bytes using the package "compress/gzip". I am writing to a bytes.Buffer and I am writing 45976 bytes, when I am trying to uncompress the content using a gzip.reader and then reader function - I find that the not all of the content is recovered. Is there some limitations to bytes.buffer? and is it a way to by pass or alter this? here is my code (edit):

func compress_and_uncompress() {
    var buf bytes.Buffer
    w := gzip.NewWriter(&buf)
    i,err := w.Write([]byte(long_string))
    if(err!=nil){
            log.Fatal(err)
    }
    w.Close()

    b2 := make([]byte, 80000)
    r, _ := gzip.NewReader(&buf)
    j, err := r.Read(b2)
    if(err!=nil){
            log.Fatal(err)
    }
    r.Close()

    fmt.Println("Wrote:", i, "Read:", j)
}

output from testing (with a chosen string as long_string) would give Wrote: 45976, Read 32768

Upvotes: 2

Views: 7499

Answers (3)

Nick Craig-Wood
Nick Craig-Wood

Reputation: 54089

Use ioutil.ReadAll. The contract for io.Reader says it doesn't have to return all the data and there is a good reason for it not to to do with sizes of internal buffers. ioutil.ReadAll works like io.Reader but will read until EOF.

Eg (untested)

import "io/ioutil"

func compress_and_uncompress() {
    var buf bytes.Buffer
    w := gzip.NewWriter(&buf)
    i,err := w.Write([]byte(long_string))
    if err!=nil {
            log.Fatal(err)
    }
    w.Close()

    r, _ := gzip.NewReader(&buf)
    b2, err := ioutil.ReadAll(r)
    if err!=nil {
            log.Fatal(err)
    }
    r.Close()

    fmt.Println("Wrote:", i, "Read:", len(b2))
}

Upvotes: 4

tredecim
tredecim

Reputation: 11

If the read from gzip.NewReader does not return the whole expected slice. You can just keep re-reading until you have recieved all the data in the buffer.

Regarding you problem where if you re-read the subsequent reads did not append to the end of the slice, but instead at the beginning; the answer can be found in the implementation of gzip's Read function, which includes

208     z.digest.Write(p[0:n])

This will result in an "append" at the beginning of the string.

This can be solves in this manner

func compress_and_uncompress(long_string string) {
    // Writer
    var buf bytes.Buffer
    w := gzip.NewWriter(&buf)
    i,err := w.Write([]byte(long_string))
    if(err!=nil){
            log.Fatal(err)
    }
    w.Close()

    // Reader
    var j, k int
    b2 := make([]byte, 80000)
    r, _ := gzip.NewReader(&buf)
    for j=0 ; ; j+=k {
        k, err = r.Read(b2[j:])  // Add the offset here
        if(err!=nil){
            if(err != io.EOF){
                log.Fatal(err)
            } else{
                break
            }
        }
    }
    r.Close()

    fmt.Println("Wrote:", i, "Read:", j)
}

The result will be:

Wrote: 45976 Read: 45976

Also after testing with a string of 45976 characters i can confirm that the output is in exactly the same manner as the input, where the second part is correctly appended after the first part.

Source for gzip.Read: http://golang.org/src/pkg/compress/gzip/gunzip.go?s=4633:4683#L189

Upvotes: 1

peterSO
peterSO

Reputation: 166598

Continue reading to get the remaining 13208 bytes. The first read returns 32768 bytes, the second read returns 13208 bytes, and the third read returns zero bytes and EOF.

For example,

package main

import (
    "bytes"
    "compress/gzip"
    "fmt"
    "io"
    "log"
)

func compress_and_uncompress() {
    var buf bytes.Buffer
    w := gzip.NewWriter(&buf)
    i, err := w.Write([]byte(long_string))
    if err != nil {
        log.Fatal(err)
    }
    w.Close()

    b2 := make([]byte, 80000)
    r, _ := gzip.NewReader(&buf)
    j := 0
    for {
        n, err := r.Read(b2[:cap(b2)])
        b2 = b2[:n]
        j += n
        if err != nil {
            if err != io.EOF {
                log.Fatal(err)
            }
            if n == 0 {
                break
            }
        }
        fmt.Println(len(b2))
    }
    r.Close()

    fmt.Println("Wrote:", i, "Read:", j)
}

var long_string string

func main() {
    long_string = string(make([]byte, 45976))
    compress_and_uncompress()
}

Output:

32768
13208
Wrote: 45976 Read: 45976

Upvotes: 7

Related Questions