How to replace file variable with gzipped version in Go?

Question

I have the following Go code:

file, err := os.Open(fileName)
if err != nil {
    fatalf(service, "Error opening %q: %v", fileName, err)
}

// Check if gzip should be applied
if *metaGzip == true {
    var b bytes.Buffer
    w := gzip.NewWriter(&b)
    w.Write(file)
    w.Close()
    file = w
}

I want to replace the file content of file with a gzipped version if metaGzip = true.

PS:
I followed this advice: Getting "bytes.Buffer does not implement io.Writer" error message but I still get the error: cannot use file (type *os.File) as type []byte in argument to w.Write

Answer 1

There are quite a few errors in your code.

As a "pre-first", always check returned errors!

First, os.Open() opens the file in read-only mode. To be able to replace the file content on the disk, you must open it in read-write mode instead:

file, err := os.OpenFile(fileName, os.O_RDWR, 0)

Next, when you open something that is an io.Closer (*os.File is an io.Closer), make sure you close it with the Close() method, best done as a deferred statement.

Next, *os.File is an io.Reader, but that is not the same thing as a byte slice []byte. An io.Reader may be used to read bytes into a byte slice. Use io.Copy() to copy the content from the file to the gzip stream (which will end up in the buffer).

In certain situation (where you don't close the gzip.Writer), you must call gzip.Writer.Flush() to ensure everything is flushed into its writer (which is the buffer in this case). Note that gzip.Writer.Close() also flushes, so this may seem like an unnecessary step, but must be done for example if the Close() of the gzip.Writer is also called as a deferred statemement, because then it may not be executed before we use the content of the buffer. Since in our examle we close the gzip writer after io.Copy(), that will take care of necessary flushes.

Next, to replace the content of the original file, you must seek back to the beginning of the file to replace. For that, you may use File.Seek().

Next, you may again use io.Copy() to copy the contents of the buffer (the gzipped data) to the file.

And last, since the gzipped content will most likely be shorter than the original file size, you must truncate the file at the size of the gzipped content (else uncompressed content of the original file may be left there).

Here's the complete code:

file, err := os.OpenFile(fileName, os.O_RDWR, 0)
if err != nil {
    log.Fatalf("Error opening %q: %v", fileName, err)
}
defer file.Close()

// Check if gzip should be applied
if *metaGzip {
    var b = &bytes.Buffer{}
    w := gzip.NewWriter(b)
    if _, err := io.Copy(w, file); err != nil {
        panic(err)
    }
    if err := w.Close(); err != nil { // This also flushes
        panic(err)
    }
    if _, err := file.Seek(0, 0); err != nil {
        panic(err)
    }
    if _, err := io.Copy(file, b); err != nil {
        panic(err)
    }
    if err := file.Truncate(int64(b.Len())); err != nil {
        panic(err)
    }
}

Note: The above code will replace the file content on your disk. If you don't want this and you just need the compressed data, you may do it like this. Note that I used a new input variable of type io.Reader, as a value of bytes.Buffer (or *bytes.Buffer) cannot be assigned to a variable of type *os.File, and we will most likely only need the result as a value of io.Reader (and this is implemented by both):

var input io.Reader

file, err := os.Open(fileName)
if err != nil {
    log.Fatalf("Error opening %q: %v", fileName, err)
}
defer file.Close()

// Check if gzip should be applied
if *metaGzip {
    var b = &bytes.Buffer{}
    w := gzip.NewWriter(b)
    if _, err := io.Copy(w, file); err != nil {
        panic(err)
    }
    if err := w.Close(); err != nil { // This also flushes
        panic(err)
    }
    input = b
} else {
    input = file
}

// Use input here

Note #2: If you don't want to "work" with the compressed data but you just want to send it e.g. as the web response, you don't even need the bytes.Buffer, you can just "stream" the compressed data to the http.ResponseWriter.

It could look like this:

func myHandler(w http.ResponseWriter, r *http.Request) {
    file, err := os.Open(fileName)
    if err != nil {
        http.NotFound(w, r)
    }
    defer file.Close()

    gz := gzip.NewWriter(w)
    defer gz.Close()

    if _, err := io.Copy(gz, file); err != nil {
        // handle error
    }
}

Proper content type will be detected and set automatically.