How to download and extract a tar file with Python

Question

Apologies for a really basic question, but I'm at my wits' end here. I cannot figure out how to download and extract an .xz tar file with Python. See the sample code below where I try multiple approaches both for downloading and for extracting, from two different tars (just to check that I'm not testing against an actually malformed tar) - all of them fail:

System info: Mac OS 10.15.7, Python 3.9.9

Code:

#!/usr/bin/env python3

import requests
import tarfile
import subprocess

from functools import partial
from shutil import copyfileobj

example_url_1 = 'https://downloads.raspberrypi.org/raspios_lite_armhf/images/raspios_lite_armhf-2022-04-07/2022-04-04-raspios-bullseye-armhf-lite.img.xz'
example_url_2 = 'https://cdimage.ubuntu.com/releases/22.04/release/ubuntu-22.04-preinstalled-server-arm64+raspi.img.xz'

def _try_extracting(filename: str):
    try:
        with tarfile.open(filename, mode='r:xz') as tf:
            tf.extractall(filename[:-3])
        print(f'Extracting {filename} succeeded')
    except Exception as e:
        print(f'Extracting {filename} failed: ', e)
        print('Falling back to extracting via command-line `tar` tool')
        p = subprocess.run(['tar', 'xf', filename], capture_output=True)
        print(p.stdout)
        print(p.stderr)
        print()
        # https://askubuntu.com/a/843803
        print('Also attempting use of command-line `ar` tool')
        p = subprocess.run(['ar', '-x', filename], capture_output=True)
        print(p.stdout)
        print(p.stderr)
        print()
        print('=========')

def _download_and_extract_in_various_ways(url, prefix):
    print('=========================================\n=========================================')
    print(f'Operating on {url}')
    print('Downloading xz via copyfileobj from requests')
    copyfileobj_download_request = requests.get(url, stream=True)
    content_length = int(copyfileobj_download_request.headers.get('content-length'))
    # https://stackoverflow.com/a/63831344/1040915
    copyfileobj_download_request.raw.read = partial(copyfileobj_download_request.raw.read, decode_content=True)
    # Here I would insert `tqdm.wrapattr(copyfileobj_download_request.raw, "read", total=content_length) as tq:`
    with open(prefix + '_tar_downloaded_with_copyfileobj.xz', 'wb') as f:
        copyfileobj(copyfileobj_download_request.raw, f) # This would be `copyfileobj(tq, f)` if fully wrapped
    _try_extracting(prefix + '_tar_downloaded_with_copyfileobj.xz')

    print('Downloading xz via bare request')
    standard_download_request = requests.get(url)
    with open(prefix + '_tar_downloaded_with_requests.xz', 'wb') as f:
        f.write(standard_download_request.content)
    _try_extracting(prefix + '_tar_downloaded_with_requests.xz')

    subprocess.run(['wget', '-O', prefix + '_tar_downloaded_with_wget.xz', url], capture_output=True)
    _try_extracting(prefix + '_tar_downloaded_with_wget.xz')

    subprocess.run(['curl', '-o', prefix + '_tar_downloaded_with_curl.xz', url], capture_output=True)
    _try_extracting(prefix + '_tar_downloaded_with_curl.xz')

def main():
    _download_and_extract_in_various_ways(example_url_1, 'raspbian')
    _download_and_extract_in_various_ways(example_url_2, 'ubuntu')


if __name__ == '__main__':
    main()

Output:

=========================================
=========================================
Operating on https://downloads.raspberrypi.org/raspios_lite_armhf/images/raspios_lite_armhf-2022-04-07/2022-04-04-raspios-bullseye-armhf-lite.img.xz
Downloading xz via copyfileobj from requests
Extracting raspbian_tar_downloaded_with_copyfileobj.xz failed:  bad checksum
Falling back to extracting via command-line `tar` tool
b''
b'tar: Error opening archive: Unrecognized archive format\n'

Also attempting use of command-line `ar` tool
b''
b'ar: raspbian_tar_downloaded_with_copyfileobj.xz: Inappropriate file type or format\n'

=========
Downloading xz via bare request
Extracting raspbian_tar_downloaded_with_requests.xz failed:  bad checksum
Falling back to extracting via command-line `tar` tool
b''
b'tar: Error opening archive: Unrecognized archive format\n'

Also attempting use of command-line `ar` tool
b''
b'ar: raspbian_tar_downloaded_with_requests.xz: Inappropriate file type or format\n'

=========
Extracting raspbian_tar_downloaded_with_wget.xz failed:  bad checksum
Falling back to extracting via command-line `tar` tool
b''
b'tar: Error opening archive: Unrecognized archive format\n'

Also attempting use of command-line `ar` tool
b''
b'ar: raspbian_tar_downloaded_with_wget.xz: Inappropriate file type or format\n'

=========
Extracting raspbian_tar_downloaded_with_curl.xz failed:  bad checksum
Falling back to extracting via command-line `tar` tool
b''
b'tar: Error opening archive: Unrecognized archive format\n'

Also attempting use of command-line `ar` tool
b''
b'ar: raspbian_tar_downloaded_with_curl.xz: Inappropriate file type or format\n'

=========
=========================================
=========================================
Operating on https://cdimage.ubuntu.com/releases/22.04/release/ubuntu-22.04-preinstalled-server-arm64+raspi.img.xz
Downloading xz via copyfileobj from requests
Extracting ubuntu_tar_downloaded_with_copyfileobj.xz failed:  bad checksum
Falling back to extracting via command-line `tar` tool
b''
b'tar: Error opening archive: Unrecognized archive format\n'

Also attempting use of command-line `ar` tool
b''
b'ar: ubuntu_tar_downloaded_with_copyfileobj.xz: Inappropriate file type or format\n'

=========
Downloading xz via bare request
Extracting ubuntu_tar_downloaded_with_requests.xz failed:  bad checksum
Falling back to extracting via command-line `tar` tool
b''
b'tar: Error opening archive: Unrecognized archive format\n'

Also attempting use of command-line `ar` tool
b''
b'ar: ubuntu_tar_downloaded_with_requests.xz: Inappropriate file type or format\n'

=========
Extracting ubuntu_tar_downloaded_with_wget.xz failed:  bad checksum
Falling back to extracting via command-line `tar` tool
b''
b'tar: Error opening archive: Unrecognized archive format\n'

Also attempting use of command-line `ar` tool
b''
b'ar: ubuntu_tar_downloaded_with_wget.xz: Inappropriate file type or format\n'

=========
Extracting ubuntu_tar_downloaded_with_curl.xz failed:  bad checksum
Falling back to extracting via command-line `tar` tool
b''
b'tar: Error opening archive: Unrecognized archive format\n'

Also attempting use of command-line `ar` tool
b''
b'ar: ubuntu_tar_downloaded_with_curl.xz: Inappropriate file type or format\n'

=========

If I download those files by directly visiting those urls in my browser, the resultant downloaded file still cannot be extracted with tar xf <file>.

Interestingly, all 8 of the files downloaded via the script ([copyfileobj,requests,wget,curl] X [raspbian,ubuntu]) can be extracted without error by "open"ing them in Mac OS Finder.

Answer 1

These files are .xz files and not tar.xz files. So Tarfile module won't work. Instead, use lzma:

import lzma
with lzma.open(filename) as fin:
    file_content = fin.read()
fout = open(filename[:-3], 'wb')
fout.write(file_content)
fout.close()