CODEWITHSUNDEEP
go
Alasdair
Alasdair

Reputation: 14161

Go: zlib uncompressing a slice of bytes

I am trying to parse a file that annoying consists of many separately zipped segments. I have parsed these segments one at a time into a slice of bytes and I want to uncompress them as I go.

Here is my current code that does the decompressing, which doesn't work. from and to are just set at the top as an example, in reality they are set by the code. data is the byte array containing the entire file. I don't want to seek it while it's on disk because its location on another server, so it's only realistic for me to load the entire file to []byte first and then parse it.

from, to := 0, 1000;
b := bytes.NewReader(data[from:from+to])
z, err := zlib.NewReader(b)
CheckErr(err)
defer z.Close()
p := make([]byte,0,1024)
z.Read(p)
fmt.Println(string(p))

So how is it so massively difficult just to unzip a slice of bytes? Anyway...

The problem appears to with how I am reading it out. Where it says z.Read, that doesn't seem to do anything.

How can I read the entire thing in one go into a slice of bytes?

Upvotes: 3

Views: 7979

Answers (2)

peterSO
peterSO

Reputation: 166935

Here's an outline for you. Note: In Go, CHECK FOR ERRORS!

package main

import (
    "bytes"
    "compress/zlib"
    "fmt"
    "io/ioutil"
)

func readSegment(data []byte, from, to int) ([]byte, error) {
    b := bytes.NewReader(data[from : from+to])
    z, err := zlib.NewReader(b)
    if err != nil {
        return nil, err
    }
    defer z.Close()
    p, err := ioutil.ReadAll(z)
    if err != nil {
        return nil, err
    }
    return p, nil
}

func main() {
    from, to := 0, 1000
    data := make([]byte, from+to)
    // ** parse input segments into data **
    p, err := readSegment(data, from, to)
    if err != nil {
        fmt.Println(err)
        return
    }
    fmt.Println(string(p))
}

Upvotes: 5

metakeule
metakeule

Reputation: 3914

Use ReadAll(r io.Reader) ([]byte, error) from the io/ioutil package.

p, err := ioutil.ReadAll(b)

fmt.Println(string(p))

Read only reads up to the length of the given slice (1024 bytes in your case).

To read in chunks of 1024 bytes:

 p := make([]byte,1024)

 for {
   numBytes, err := l.Read(p)
   if err == io.EOF {
      // you are done, numBytes might be less than len(p)
      break
   }
   // do what you want with p
 }

If you are getting the data from a webserver, you might even do

 import (
   "net/http"
   "io/ioutil"
 )
 ...
 resp, errGet := http.Get("http://example.com/somefile")
 // do error handling
 z, errZ := zlib.NewReader(resp.Body)
 // do error handling
 resp.Body.Close()
 p, err := ioutil.ReadAll(b)
 // do error handling

since resp.Body happens to be an io.Reader as most io related types.

Upvotes: 3

Related Questions