UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 73776

Question

This is a Django project. Encountering error upon running pip install -r requirements.txt in a local virtualenv.

Collecting https://github.com/jedestep/Zappa/archive/layer-support.zip (from -r requirements\base.txt (line 9))
Using cached https://github.com/jedestep/Zappa/archive/layer-support.zip
ERROR: Command errored out with exit status 1:
     command: 'c:\users\user~1\desktop\project\project\venv\scripts\python.exe' -c 'import sys, setuptools, tokenize
    ; sys.argv[0] = '"'"'C:\\Users\\USER~1\\AppData\\Local\\Temp\\pip-req-build-6htw2gh2\\setup.py'"'"'; __file__='"'"'C:\
    \Users\\USER~1\\AppData\\Local\\Temp\\pip-req-build-6htw2gh2\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(
    __file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' e
    gg_info --egg-base 'C:\Users\USER~1\AppData\Local\Temp\pip-req-build-6htw2gh2\pip-egg-info'
         cwd: C:\Users\USER~1\AppData\Local\Temp\pip-req-build-6htw2gh2\
    Complete output (7 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "C:\Users\USER~1\AppData\Local\Temp\pip-req-build-6htw2gh2\setup.py", line 8, in <module>
        long_description = readme_file.read()
      File "c:\users\user~1\desktop\project\project\venv\lib\encodings\cp1252.py", line 23, in decode
        return codecs.charmap_decode(input,self.errors,decoding_table)[0]
    UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 73776: character maps to <undefined>
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

The requirements do install as expected in the vagrant environment, however I recall that it would also work locally on my machine. Unfortunately, I'm not sure what was the change that lead to this error, it may have been the result of a pull.

Versions: Python: 3.6.5 Django: 2.2 pip: 20.0.2

I have run pip install --upgrade setuptools but it does not change the error. A similar error will occur with the automatic test database downloads when I try to run tox.

Any ideas would be appreciated. Thanks.

Answer 1

The requirements do install as expected in the vagrant environment, however I recall that it would also work locally on my machine. Unfortunately, I'm not sure what was the change that lead to this error, it may have been the result of a pull.

The issue is with the package you're trying to install, and the configuration of your environment. I guess the package updated at one point and broke because they added emoji to their readme or something: if you look a bit at the error message you can see there's a problem in reading stuff into long_description, check the package's setup.py and you can find the issue near the start:

with open('README.md') as readme_file:
    long_description = readme_file.read()

The problem here is that when you open a file without specifying a mode it's going to be "text" so python will automatically decode bytes to str, which is cool, except the encoding it uses for such decoding is the "default" one for the system (found by calling locale.getpreferredencoding(False)), which often is not what's desirable.

Your Vagrant environment and the developer's machine probably have a UTF8 default encoding (as most unix systems would these days), meanwhile your local windows box does not and uses CP1252 by default which blows up. Sadly I'm not much of a windows user (for dev) so I've no idea how to change the "default encoding", that may even not be possible given what a cursory search of SO yields.

I would suggest:

• asking the package author to fix their setup.py to explicitly specify an encoding when opening file
• trying to find out more about getpreferredencoding's internal decisions, and whether it's possible to override the preferredencoding via the environment in Windows (it's apparently possible / easy to override the encoding for the standard stream but chcp / set PYTHONIOENCODING apparently don't do anything for a normal open)
• running your thing on the Windows Subsystem for Linux, which would probably work out of the box

PS: I've taken the liberty of updating your post's tags as django, pip and eggs really have nothing to do with the actual issue