dcode
dcode

Reputation: 33

How do I upload qcow2 images to libvirt storage using the API and keep compression?

Using the libvirt-python module, I tried creating the volume with XML using the information provided from the output of qemu-img info. Then I upload using a stream. This works and the disk is functional, but libvirt allocates the physical size equal to the virtual size.

import libvirt 
import pathlib

conn = libvirt.open("qemu:///system")
template_pool = conn.storagePoolLookupByName("default")
image_filename = "fedora-cloud-base-38-x86_64.qcow2"
spool_file = pathlib.Path("Fedora-Cloud-Base-38-1.6.x86_64.qcow2")

# Size from output of `qemu-img info --output=json Fedora-Cloud-Base-38-1.6.x86_64.qcow2`
image_size = 497291264
image_vsize = 5368709120
stgvol_xml = f"""
<volume>
  <name>{image_filename}</name>
  <allocation>{ image_size }</allocation>
  <capacity unit="bytes">{image_vsize}</capacity>
</volume>"""

stgvol = template_pool.createXML(stgvol_xml, 0)

def stream_handler(stream, data, file_):
    return file_.read(data)

stream = conn.newStream(0)
stgvol.upload(stream, 0, 0)
with spool_file.open(mode="rb") as f:
    stream.sendAll(stream_handler, f)
    stream.finish()

In my tests, I'm using the Fedora Cloud Base 38 qcow2 image from the download site. After a fresh download, the SHA256 hash verifies and the allocated size is 475MB, aligning with qemu-img info output

> ls -lsh Fedora-Cloud-Base-38-1.6.x86_64.qcow2
475M -rw-r--r-- 1 dcode primarygroup 475M Jun  3 10:05 Fedora-Cloud-Base-38-1.6.x86_64.qcow2

> sha256sum Fedora-Cloud-Base-38-1.6.x86_64.qcow2
d334670401ff3d5b4129fcc662cf64f5a6e568228af59076cc449a4945318482  Fedora-Cloud-Base-38-1.6.x86_64.qcow2

> qemu-img info Fedora-Cloud-Base-38-1.6.x86_64.qcow2 
image: Fedora-Cloud-Base-38-1.6.x86_64.qcow2
file format: qcow2
virtual size: 5 GiB (5368709120 bytes)
disk size: 474 MiB
cluster_size: 65536
Format specific information:
    compat: 0.10
    compression type: zlib
    refcount bits: 16

In the data store, however, the file is expanded to the full "virtual size" as a sparse image. Of course, this changes the hash value, but the image info remains the same.

# ls -lsh Fedora-Cloud-Base-38-1.6.x86_64.qcow2 
475M -rw-------. 1 root root 5.0G Jun  3 15:10 Fedora-Cloud-Base-38-1.6.x86_64.qcow2

# sha256sum Fedora-Cloud-Base-38-1.6.x86_64.qcow2
2327ccac1781e9ff561737eb986f9863ff65827fe53efd565e6c2b801273625f  Fedora-Cloud-Base-38-1.6.x86_64.qcow2

# qemu-img info Fedora-Cloud-Base-38-1.6.x86_64.qcow2 
image: Fedora-Cloud-Base-38-1.6.x86_64.qcow2
file format: qcow2
virtual size: 5 GiB (5368709120 bytes)
disk size: 474 MiB
cluster_size: 65536
Format specific information:
    compat: 0.10
    compression type: zlib
    refcount bits: 16

As a workaround, I've run virt-sparsify on the resulting disk with the --compress option, which does reduce the size back to somewhere close to the original, but I haven't been able to reproduce the original file with matching hashes. And that's not a great solution anyway because the code I'm using may not have local access to the libvirt datastore.

I also tried mucking with the sparsestream.py example from libvirt-python, but it's not super clear to me of what's happening, and this doesn't seem to be a sparse issue, but rather compression.

Upvotes: 1

Views: 494

Answers (0)

Related Questions