Why does `vagrant box add` yield a `bsdtar` error importing a `.box` in Ubuntu?

Question

I installed vagrant and virtual box in Ubuntu 12.04.2 LTS. (Vagrant version 1.2.1). I have the vagrant box downloaded and executed the vagrant box add command. I am getting the following error:

*vagrant box add base64 package.box

Downloading or copying the box...
Extracting box...te: 147M/s, Estimated time remaining: 0:00:01)
The box failed to unpackage properly. Please verify that the box
file you're trying to add is not corrupted and try again. The
output from attempting to unpackage (if any):
x ./box-disk1.vmdk: Write failed
x ./box.ovf: Write failed
x ./Vagrantfile: Write failed
bsdtar: Error exit delayed from previous errors.*

I repeated the same with other versions of vagrant(like version 1.2.2, 1.0.1, 1.3.0), but the same error. I am sure that the vagrant box I am adding is not corrupt, as I have used the same in mac machine and it worked fine...

Answer 1

It seems like the .box file isn't corrupted but your vagrant needs write permissions. I think you should check the installation of vagrant.

Answer 2

Problem / Explanation

I am sure that the vagrant box I am adding is not corrupt, as I have used the same in mac machine and it worked fine...

I have been able to reproduce this issue on various occasions by doing the following:

1. Create a Virtualbox VM in macOS (Intel-based mac running anything... Mojave, Catalina, Mojave, Big Sur, etc...)

2. vagrant package --base $VIRTUALBOX_VM_NAME_HERE --output test.box

3. Transfer the .box file over to Ubuntu

4. vagrant box add --provider virtualbox --name $VAGRANT_BOX_NAME_HERE test.box

5. Unpack error:

    The box failed to unpackage properly. Please verify that the box
    file you're trying to add is not corrupted and that enough disk space
    is available and then try again.
    The output from attempting to unpackage (if any):
   
    x ./include/
    x ./include/README.md
    x ./include/_Vagrantfile
    x ./include/LICENSE
    x ./metadata.json
    x ./box.ovf
    x ./Vagrantfile
    x ./info.json
    x ./box-disk001.vmdk: gzip decompression failed
    bsdtar: Error exit delayed from previous errors.
   

The answer lies in the hidden context behind the error message:

bsdtar: Error exit delayed from previous errors.*

Notice that this vagrant box add ... was run on Ubuntu, and is trying to unpack using bsdtar, rather than the usual tar command. The .box file was created on macOS and compressed with its own version of bsdtar. Usually, native .tar files on Linux are created using the GNU version of tar. There are some important differences between bsdtar, GNU tar, and the Linux version libarchive bsdtar.

This means that we cannot expect in all cases to create a .tar file on one platform and be able to unpack it on another platform without the appropriate compatible version of tar which can handle that file. The file is not corrupt, it's just in a different format than the Linux version of bsdtar is expecting to handle.

One could make the argument that vagrant package is intended to create a portable .box file. However, thanks to the differences between classic bsdtar, libarchive bsdtar, and GNU tar, we end up with a non-portability problem. Virtual Machine disk files are commonly implemented using sparse files. Sparse files are used by VirtualBox for the .vmdk file format to support growing and shrinking the Guest VM disk dynamically while still remaining relatively small on the host OS. Depending on your version, bsdtar handles these sparse files differently than other versions of tar. Vagrant uses bsdtar, but this does not ensure that the resulting .box file is compatible and fully portable across systems. MacOS ships with a different version of bsdtar than is currently available on Ubuntu Linux.

macOS Big Sur

$ sw_vers
ProductName:    macOS
ProductVersion: 11.6.1
BuildVersion:   20G224
$ ls -l /usr/bin/tar
lrwxr-xr-x  1 root  wheel  6 Jan  1  2020 /usr/bin/tar -> bsdtar
$ bsdtar --version
bsdtar 3.3.2 - libarchive 3.3.2 zlib/1.2.11 liblzma/5.0.5 bz2lib/1.0.6

Ubuntu Focal Fossa

$ lsb_release -a
Distributor ID: Ubuntu
Description:    Ubuntu 20.04.3 LTS
Release:        20.04
Codename:       focal
$ tar --version
tar (GNU tar) 1.30
Copyright (C) 2017 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by John Gilmore and Jay Fenlason.
$ apt-cache show libarchive-tools | grep 'Description-en'
Description-en: FreeBSD implementations of 'tar' and 'cpio' and other archive tools
$ sudo apt-get install libarchive-tools
$ bsdtar --version
bsdtar 3.4.0 - libarchive 3.4.0 zlib/1.2.11 liblzma/5.2.4 bz2lib/1.0.8 liblz4/1.9.2 libzstd/1.4.4

Solution

One workaround I've found for this issue is to avoid using the .box file created by vagrant package on macOS. Unless you're also importing to another macOS machine, this .box file will not work thanks to the bsdtar issue. Instead, copy the files separately from within $HOME/.vagrant.d/boxes/$VAGRANT_BOX_NAME_HERE. These are the bare VirtualBox .ovf and .vmdk files that should be compatible with VirtualBox on Linux too. Once these files are safely transferred to Linux via whatever method you prefer, they can be loaded into VirtualBox there or used directly from .vagrant.d/boxes via vagrant commands.

You have a few options to transfer the files. I'd suggest using rsync to avoid issues with tar & bsdtar.

# From macOS machine
rsync --progress -av ~/.vagrant.d/boxes/someuser-VAGRANTSLASH-vagrant-box-example-name  target-machine.local:~/.vagrant.d/boxes/
# Or... from Linux machine side
rsync --progress -av your-mbp.local:~/.vagrant.d/boxes/someuser-VAGRANTSLASH-vagrant-box-example-name  ~/.vagrant.d/boxes/

Note: Slashes / matter in the rsync source and target paths! See this brief rsync tutorial for help using rsync

Once your files have transferred, be sure to check that they are identical. I use shasum -a 256 to generate SHA256 hashes for the files. The hashes should match on both sides.

# Check on both machines separately...
cd  ~/.vagrant.d/boxes/someuser-VAGRANTSLASH-vagrant-box-example-name/0/virtualbox/
shasum -a 256 box-disk001.vmdk
# Or... use `shasum -c -` and SSH for a one-liner:
# Prints "OK" and exit status 0 if checksums match
( cd  ~/.vagrant.d/boxes/someuser-VAGRANTSLASH-vagrant-box-example-name/0/virtualbox/  && shasum -a 256 box-disk001.vmdk )  | ssh  exampleuser@your-mbp.local '( cd ~/.vagrant.d/boxes/someuser-VAGRANTSLASH-vagrant-box-example-name/0/virtualbox/ && shasum -a 256 -c - )'

If you really want to try using [bsd]tar or .box files...

Another option is using gnu-tar from Homebrew to repackage the files as a compatible .tar archive. Then copy that over and unpack into .vagrant.d/boxes. I tried this and found it still resulted in a non-identical .vmdk disk file on the Linux side. (Check this on both machines with sha256sum or perl's shasum -a 256).

Yet another option is to use bsdtar from libarchive-tools package on the Ubuntu machine. I tried this and was able to reproduce the same error from vagrant box add

# View .box file contents
bsdtar -v -t  -f /path/to/Vagrant_Box/example.box
# Try to extract
mkdir ~/.vagrant.d/boxes/someuser-VAGRANTSLASH-vagrant-box-example-name
cd ~/.vagrant.d/boxes/someuser-VAGRANTSLASH-vagrant-box-example-name
bsdtar -v -x  -f /path/to/Vagrant_Box/example.box
x ./include/
x ./include/README.md
x ./include/_Vagrantfile
x ./include/LICENSE
x ./metadata.json
x ./box.ovf
x ./Vagrantfile
x ./info.json
x ./box-disk001.vmdk: gzip decompression failed
bsdtar: Error exit delayed from previous errors.

So, it's best to avoid using tar and bsdtar across these platforms because of their incompatible handling of the box-disk001.vmdk sparse file.

Vagrant Box Format Contents

As we can see above, an example .box unpacked structure might be something like the following:

/Users/exampleuser/.vagrant.d/boxes/someuser-VAGRANTSLASH-vagrant-box-example-name/
└── 0
    └── virtualbox
        ├── Vagrantfile
        ├── box-disk001.vmdk
        ├── box.ovf
        ├── include
        │   ├── LICENSE
        │   ├── README.md
        │   └── _Vagrantfile
        ├── info.json
        └── metadata.json

3 directories, 8 files

Note: The info.json file and anything under include (e.g.: README.md, _Vagrantfile) are optional. These can be added via vagrant package options: --vagrantfile, --info, and --include. They are not necessary to run the VM.

These files can be dropped into the Ubuntu machine's ~/.vagrant.d/boxes/ folder in that same structure. Make sure the box directory name matches your intended box name (e.g.: directory someuser-VAGRANTSLASH-vagrant-box-example-name = box name someuser/vagrant-box-example-name).

Set up a Vagrantfile to run the box using the box name, then run vagrant up as usual.

Example:

Vagrant.configure("2") do |c|
  c.berkshelf.enabled = false if Vagrant.has_plugin?("vagrant-berkshelf")
  c.vm.box = "someuser/vagrant-box-example-name"
  c.vm.hostname = "default-vagrant-box-example-name.vagrantup.com"
  c.vm.boot_timeout = 1200
  c.vm.synced_folder ".", "/vagrant", disabled: true
  c.vm.provider :virtualbox do |p|
    p.name = "vagrant-box-example-name"
    p.customize ["modifyvm", :id, "--audio", "none"]
  end
end

macOS VM Notes

If the .box file in question is a macOS VM, then you may still not be able to run it on Linux easily. The reason is that Apple's boot.efi has macOS-specific boot firmware. Technically it's not supported to run macOS on non-Apple hardware (Ya know... because Capitalism... 💸😒). I'm sure someone somewhere has figured it out given that creating hardware Hackintosh machines is possible. I haven't yet found a way to get this working, but maybe someone else knows... 🤷

Answer 3

I had the same problem. When you see write failed it usually means you ran out of disk space. The answer is that you need space twice the size of the actual file on your HDD. During packaging, Vagrant requires 2x the original space since it copies all the files.

Answer 4

I have the same problem to fix this you need to add the url source from http://www.vagrantbox.es/ search for your OS (Devopera Ubuntu 12.04 LAMP stack) if you use LAMP.

sudo vagrant box add laravel/homestead http://devopera.com/node/63/download/ubuntu1204/dobu12-lamp-vagrant.box