Uncompressing a .Z file with Python

Question

I'm trying to uncompress a *.Z file using Python. I downloaded it via FTP (binary mode). The file successfully uncompresses with 7zip (whose "info" on the file says it's of type "Z"). The original file can be found at ftp://cddis.gsfc.nasa.gov/gps/products/1860/igr18600.sp3.Z.

I've read up on the use of the zlib module in Python and have some test code I'm using:

import zlib

comp_data = open('C:\Temp\igr18600.sp3.Z', 'rb').read()

print(comp_data[0:10])

uncomp_data = zlib.decompress(comp_data)
with open('c:\temp\igr18600.sp3', 'wb') as f:
    f.write(uncomp_data)
    f.close()

When I execute this I get the following output:

b'\x1f\x9d\x90#\xc6@\x91\x01#F'
Traceback (most recent call last):
  File "test.py", line 7, in <module>
    uncomp_data = zlib.decompress(comp_data)
zlib.error: Error -3 while decompressing data: incorrect header check

zlib clearly doesn't like the header. The first couple of bytes appear to match the proper magic number sequence 0x1F9d for a compressed file (per https://en.wikipedia.org/wiki/List_of_file_signatures).

In a pinch I can work around this by shelling out to 7zip directly. But I was hoping to find a pure Python type of answer. Despite spending most of the day googling for an answer (or this error message) I haven't had much luck. Perhaps my search skills are atrophying?

Answer 1

Some years have gone by, and now there are python packages for this: unlzw and unlzw3^[1]

1) unlzw

• GitHub: ionelmc/python-unlzw (Last commit 2017-10)
• PyPI: unlzw
• Docs: python-unlzw.readthedocs.io
• Pros/Cons: The package provides a C-extension file (.so or .pyd) which makes it fast. The downside is that it has built versions only up to CPython 3.6 (as it is not maintained?), and if you install it on Windows, pip will download the .tar.gz, and you'll have to have a C-compiler installed to make the installation to work properly. I tried and since I have a C-compiler installed, got it working with CPython 3.8.6.

1.1) Installation*

pip install unlzw

^{*See the note on Pros/Cons above.}

1.2) Example usage

from unlzw import unlzw

with open('file.Z', 'rb') as fh:
    compressed_data = fh.read()
    uncompressed_data = unlzw(compressed_data)

2) unlzw3

• GitHub: scivision/unlzw3 (Last commit 2020-07). Based on github.com/umeat/unlzw
• PyPI: unlzw3
• Pros/Cons: Pure python implementation, meaning that it will install & run on "any" platform with python, including Windows without C-compiler. It is also slower than the C-implementation (see benchmarks below).

2.1) Installation

pip install unlzw3

2.2) Example usage

From the documentation:

import unlzw3
from pathlib import Path

uncompressed_data = unlzw3.unlzw(Path('file.Z'))

Appendix: Benchmarks

Using a file igsg1450.20i.Z downloaded from https://cddis.nasa.gov/archive/gnss/products/ionex/2020/145/.

unlzw
3.5 ms ± 93 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

unlzw3
166 ms ± 2.7 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

The benchmarks show, that for 1000 similar .Z files you would have to wait only 3.5 seconds with unlzw, but 2.8 minutes with unlzw3. For one-off conversion, the difference probably does not matter.

---

[1] All of these are actually based on the Mark Adler's answer on mathematica.stackexchange.com

Answer 2

Python does not have the equivalent of Unix uncompress available in a module, which is what you'd need to decompress a .Z file. You would either need to a) shell out to the Unix compress command, b) shell out to gzip, c) shell out to 7-zip (both gzip and 7-zip have the ability to decompress .Z files), d) modify the original uncompress code in C and link that to Python (the code is available online), or e) write your own LZW decompressor in native Python.

For d), you can find some C code I wrote to do this job in this answer on mathematica.stackexchange.com. See the unlzw() function.