CODEWITHSUNDEEP
go
Jeff
Jeff

Reputation: 7210

Golang: append file to an existing tar archive

How would I append a file to an existing tar archive in Go? I don't see anything obvious in the docs on how to do it.

I have a tar file that has already been created and I want to add more to it after it has already been closed.

EDIT

Altering the example in the docs and following the answer given, I'm still not getting the expected result. The first three files are being written to the tar but when I close and open up the file again to write to it, the new file is never being written. The code runs fine. I don't know what I'm missing.

The following code gives me a tar file with three files in it: readme.txt, gopher.txt, todo.txt. foo.bar never gets written.

package main

import (
    "archive/tar"
    "log"
    "os"
)

func main() {
    f, err := os.Create("/home/jeff/Desktop/test.tar")
    if err != nil {
        log.Fatalln(err)
    }

    tw := tar.NewWriter(f)

    var files = []struct {
        Name, Body string
    }{
        {"readme.txt", "This archive contains some text files."},
        {"gopher.txt", "Gopher names:\nGeorge\nGeoffrey\nGonzo"},
        {"todo.txt", "Get animal handling licence."},
    }
    for _, file := range files {
        hdr := &tar.Header{
            Name: file.Name,
            Size: int64(len(file.Body)),
        }
        if err := tw.WriteHeader(hdr); err != nil {
            log.Fatalln(err)
        }
        if _, err := tw.Write([]byte(file.Body)); err != nil {
            log.Fatalln(err)
        }
    }
    if err := tw.Close(); err != nil {
        log.Fatalln(err)
    }
    f.Close()

    // Open up the file and append more things to it

    f, err = os.OpenFile("/home/jeff/Desktop/test.tar", os.O_APPEND|os.O_WRONLY, os.ModePerm)
    if err != nil {
        log.Fatalln(err)
    }
    tw = tar.NewWriter(f)

    test := "this is a test"

    hdr := &tar.Header{
        Name: "foo.bar",
        Size: int64(len(test)),
    }

    if err := tw.WriteHeader(hdr); err != nil {
        log.Fatalln(err)
    }

    if _, err := tw.Write([]byte(test)); err != nil {
        log.Fatalln(err)
    }

    if err := tw.Close(); err != nil {
        log.Fatalln(err)
    }
    f.Close()

}

Upvotes: 8

Views: 7551

Answers (3)

Matt
Matt

Reputation: 23759

I found the accepted solution by @Intermernet to be basically correct, except that the padding at the end of an archive is generally arbitrary (unless you control the writer too).

This works for me as far as I can tell so far:

var lastFileSize, lastStreamPos int64
tr := tar.NewReader(output)
for {
    hdr, err := tr.Next()
    if err == io.EOF {
        break
    }
    if err != nil {
        return err
    }
    lastStreamPos, err = output.Seek(0, io.SeekCurrent)
    if err != nil {
        return err
    }
    lastFileSize = hdr.Size
}

const blockSize = 512
newOffset := lastStreamPos + lastFileSize
// shift to next-nearest block boundary (unless we are already on it)
if (newOffset % blockSize) != 0 {
    newOffset += blockSize - (newOffset % blockSize) 
}
_, err := output.Seek(newOffset, io.SeekStart)
if err != nil {
    return err
}

tw := tar.NewWriter(output)
defer tw.Close()

// ... proceed to write to tw as usual ...

I'm not 100% sure why this works. But the basic idea is to scan the archive, find the offset of the start of the last file, then skip over it, then align to the nearest block boundary, then start overwriting from there.

Upvotes: 0

Intermernet
Intermernet

Reputation: 19388

The tar file specification states:

A tar archive consists of a series of 512-byte records. Each file system object requires a header record which stores basic metadata (pathname, owner, permissions, etc.) and zero or more records containing any file data. The end of the archive is indicated by two records consisting entirely of zero bytes.

The Go implementation of adding these two zero filled records happens here .

To get around the tar file format trailer (basically 1024 bytes of nothing) you could replace the lines:

f, err = os.OpenFile("/home/jeff/Desktop/test.tar", os.O_APPEND|os.O_WRONLY, os.ModePerm)
if err != nil {
    log.Fatalln(err)
}
tw = tar.NewWriter(f)

With:

f, err = os.OpenFile("/home/jeff/Desktop/test.tar", os.O_RDWR, os.ModePerm)
if err != nil {
    log.Fatalln(err)
}
if _, err = f.Seek(-1024, os.SEEK_END); err != nil {
    log.Fatalln(err)
}
tw = tar.NewWriter(f)

It opens the file read / write (instead of append / write-only) and then seeks to 1024 bytes before the end of the file and writes from there.

It works, but it is a horrible hack.

EDIT: After understanding the tar file spec a little better, I no longer believe this is such a hack.

Full code: http://play.golang.org/p/0zRScmY4AC

Upvotes: 14

Jeremy Wall
Jeremy Wall

Reputation: 25245

It's just a writer interface so write bytes to it after writing your files header.

import (
  "archive/tar"
  "os"
)

f, err := os.OpenFile(path, os.O_APPEND|os.O_WRONLY, os.ModePerm)
if err != nil {
// handle error here
}

hdr := tar.Header{}
// populate your header
tw := tar.NewWriter(f)
// append a file
tw.WriteHeader(hdr)
tw.Write(content_of_file_as_bytes)

http://golang.org/pkg/archive/tar/#Writer tells you all you need to know.

EDIT: It turns out that tar files get a trailer written to the end when it's closed. So even though you are writing new data to the tar archive it won't be read past that trailer. So instead it looks like you'll have to read in the tar archive first and then rewrite the whole archive to disk which is suboptimal. The package doesn't support the necessary stuff to append to them though so that's the best I can recommend right now.

Upvotes: 4

Related Questions