hyperwiser
hyperwiser

Reputation: 497

Python zipfile module can't extract filenames with Chinese characters

I'm trying to use a python script to download files from a Chinese service provider (I'm not from China myself). The provider is giving me a .zip file which contains a file which seems to have Chinese characters in its name. This seems to be causing the zipfile module to barf.

Code:

import zipfile

f = "/path/to/zip_file.zip"

if zipfile.is_zipfile(f):
    fz = zipfile.ZipFile(f, 'r')

The zipfile itself doesn't contain any non-ASCII characters but the file inside it does. When I run the above script i get the following exception:

Traceback (most recent call last):   File "./temp.py", line 9, in <module>
    fz = zipfile.ZipFile(f, 'r')   File "/usr/lib/python2.7/zipfile.py", line 770, in __init__
    self._RealGetContents()   File "/usr/lib/python2.7/zipfile.py", line 859, in _RealGetContents
    x.filename = x._decodeFilename()   File "/usr/lib/python2.7/zipfile.py", line 379, in _decodeFilename
    return self.filename.decode('utf-8')   File "/usr/lib/python2.7/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True) UnicodeDecodeError: 'utf8' codec can't decode byte 0xbd in position 30: invalid start byte

I've tried looking through the answers to many similar questions:

Please correct me if I'm wrong, but it looks like an open issue with the zipfile module.

How do I get around this? Is there any alternative module for dealing with zipfiles that I should use? Or any other solution?

TIA.

Edit: I can access/unzip the same file perfectly with the linux command-line utility "unzip".

Upvotes: 11

Views: 12208

Answers (8)

jianguo_xiao
jianguo_xiao

Reputation: 1

In my case,i solved the problem by add parameter metadata_encoding='utf-8' to the ZipFile function.

f = zipfile.ZipFile("./dataset_delete_test.zip",'r',metadata_encoding='utf-8')

Through a quick debug,i found that there was an exception in an if statement in a ZipFile that evaluates filename to cp437. The reason is that flags is 0

#this code will lead filename.decode='cp437' in class ZipFile of zipfile.py

f = zipfile.ZipFile("./dataset_delete_test.zip",'r')

the if statement about the filename's decode in the zipfile.py

 if flags & _MASK_UTF_FILENAME:
            # UTF-8 file names extension
            filename = filename.decode('utf-8')
        else:
            # Historical ZIP filename encoding
            filename = filename.decode(self.metadata_encoding or 'cp437')

Upvotes: 0

Qingxu Fu
Qingxu Fu

Reputation: 1

In my opinion, this is a better solution to the previous answers.

Change:

        with zipfile.ZipFile(file_path, "r") as zipobj:
            zipobj.extractall(path=dest_dir)
            print("Successfully extracted zip archive to {}".format(dest_dir))

to:

        with zipfile.ZipFile(file_path, "r") as zipobj:
            zipobj._extract_member = lambda a,b,c: _extract_member_new(zipobj, a,b,c)
            zipobj.extractall(path=dest_dir)
            print("Successfully extracted zip archive to {}".format(dest_dir))

where _extract_member_new is:

def _extract_member(self, member, targetpath, pwd):
    """Extract the ZipInfo object 'member' to a physical
        file on the path targetpath.
    """
    import zipfile
    if not isinstance(member, zipfile.ZipInfo):
        member = self.getinfo(member)

    # build the destination pathname, replacing
    # forward slashes to platform specific separators.
    arcname = member.filename.replace('/', os.path.sep)
    arcname = arcname.encode('cp437', errors='replace').decode('gbk', errors='replace')

    if os.path.altsep:
        arcname = arcname.replace(os.path.altsep, os.path.sep)
    # interpret absolute pathname as relative, remove drive letter or
    # UNC path, redundant separators, "." and ".." components.
    arcname = os.path.splitdrive(arcname)[1]
    invalid_path_parts = ('', os.path.curdir, os.path.pardir)
    arcname = os.path.sep.join(x for x in arcname.split(os.path.sep)
                                if x not in invalid_path_parts)
    if os.path.sep == '\\':
        # filter illegal characters on Windows
        arcname = self._sanitize_windows_name(arcname, os.path.sep)

    targetpath = os.path.join(targetpath, arcname)
    targetpath = os.path.normpath(targetpath)

    # Create all upper directories if necessary.
    upperdirs = os.path.dirname(targetpath)
    if upperdirs and not os.path.exists(upperdirs):
        os.makedirs(upperdirs)

    if member.is_dir():
        if not os.path.isdir(targetpath):
            os.mkdir(targetpath)
        return targetpath

    with self.open(member, pwd=pwd) as source, \
            open(targetpath, "wb") as target:
        shutil.copyfileobj(source, target)

    return targetpath

Upvotes: -1

Victor Stafusa
Victor Stafusa

Reputation: 14613

This is almost 6 years late, but this was finally fixed in Python 3.11 with the addition of the metadata_encoding parameter. I posted this answer here anyway to help other people with similar issues.

import zipfile

f = "your/zip/file.zip"
t = "the/dir/where/you/want/to/extract/it/all"

with zipfile.ZipFile(f, "r", metadata_encoding = "utf-8") as zf:
    zf.extractall(t)

Upvotes: 7

SK-415
SK-415

Reputation: 11

@Mr.Ham's solution perfectly solved my problem. I'm using the Chinese version of Win10. Which the default encoding of the file system is GBK.

I think for other language users. Just change decode from GBK to their system default encoding will also work. And the default system encoding could automaticly get by Python.

So the patched code looks like this:

import zipfile
import locale

default_encoding = locale.getpreferredencoding()

with zipfile.ZipFile("/path/to/zip_file.zip") as f:
    zipinfo = f.infolist()
    for member in zipinfo:
        member.filename = member.filename.encode('cp437').decode(default_encoding)
        # The second argument could make the extracted filese to the same dir as the zip file, or leave it blank to your work dir.
        f.extract(member, "/path/to/zip_file")

Upvotes: 1

secsilm
secsilm

Reputation: 430

Recently I met the same problem. Here is my solution. I hope it is useful for you.

import shutil
import zipfile
f = zipfile.ZipFile('/path/to/zip_file.zip', 'r')
for fileinfo in f.infolist():
    filename = fileinfo.filename.encode('cp437').decode('gbk')
    outputfile = open(filename, "wb")
    shutil.copyfileobj(f.open(fileinfo.filename), outputfile)
    outputfile.close()
f.close()

UPDATE: You can use the following simpler solution with pathlib:

from pathlib import Path
import zipfile

with zipfile.ZipFile('/path/to/zip_file.zip', 'r') as f:
    for fn in f.namelist():
        extracted_path = Path(f.extract(fn))
        extracted_path.rename(fn.encode('cp437').decode('gbk'))

Upvotes: 8

Mr.Ham
Mr.Ham

Reputation: 31

What about this code?

import zipfile

with zipfile.ZipFile('/path/to/zip_file.zip', 'r') as f:
    zipInfo = f.infolist()
    for member in zipInfo:
        member.filename = member.filename.encode('cp437').decode('gbk')
        f.extract(member)

Upvotes: 2

socrates
socrates

Reputation: 1321

The way of Python 2.x(2.7) and Python 3.x dealing with non utf-8 filename in module zipfile are a bit different.

First, they both check ZipInfo.flag_bits of the file, if ZipInfo.flag_bits & 0x800, name of the file will be decode with utf-8.

If the check of above is False, in Python 2.x, the byte string of the name will be returned; in Python 3.x, the module will decode the file with encoding cp437 and return decoded result. Of course, the module will not know the true encoding of the filename in both Python versions.

So, suppose you have got a filename from a ZipInfo object or zipfile.namelist method, and you have already know the filename is encoded with XXX encoding. Those are the ways you get the correct unicode filename:

# in python 2.x
filename = filename.decode('XXX')


# in python 3.x
filename = filename.encode('cp437').decode('XXX')

Upvotes: 16

bobince
bobince

Reputation: 536567

The ZIP file is invalid. It has a flag that signals that filenames inside it are encoded as UTF-8, but they're actually not; they contain byte sequences that aren't valid as UTF-8. Maybe they're GBK? Maybe something else? Maybe some unholy inconsistent mixture? ZIP tools in the wild are unfortunately very very poor at handling non-ASCII filenames consistently.

A quick workaround might be to replace the library function that decodes the filenames. This is a monkey-patch as there isn't a simple way to inject your own ZipInfo class into ZipFile, but:

zipfile.ZipInfo._decodeFilename = lambda self: self.filename

would disable the attempt to decode the filename, and always return a ZipInfo with a byte string filename property that you can proceed to decode/handle manually in whatever way is appropriate.

Upvotes: 1

Related Questions