CODEWITHSUNDEEP
utf-8cmdpipepython-3.5pyinstaller
Bigfoot29
Bigfoot29

Reputation: 70

Windows cmd: piping python 3.5 py file results works but pyinstaller exe's leads to UnicodeEncodeError

I am somewhat out of options here...

# -*- coding: utf-8 -*-
print(chr(246) + " " + chr(9786) + " " + chr(9787))
print("End.")

When I run the code mentioned above in my Win7 cmd window, I get the results depending on the way I invoke it:

python.exe utf8.py
-> ö ☺ ☻

python.exe utf8.py >test.txt
-> ö ☺ ☻ (in file)

utf8.exe
-> ö ☺ ☻

utf8.exe >test.txt
RuntimeWarning: sys.stdin.encoding == 'utf-8', whereas sys.stdout.encoding == 'cp1252', readline hook consumer may assume they are the same
Traceback (most recent call last):
  File "Development\utf8.py", line 15, in <module>
    print(chr(246) + " " + chr(9786) + " " + chr(9787))
  File "C:\python35\lib\encodings\cp1252.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode character '\u263a' in position

Messing around with win_unicode_console doesn't help either. In the end, I get the same results.

PYTHONIOENCODING=utf-8

is set. But it seems, that when using PyInstaller, the parameter is ignored for stdout.encoding:

print(sys.stdout.encoding)
print(sys.stdout.isatty())
print(locale.getpreferredencoding())
print(sys.getfilesystemencoding())
print(os.environ["PYTHONIOENCODING"])

Output:

python.exe utf8.py > test.txt
utf-8
False
cp1252
mbcs
utf-8

utf8.exe >test.txt
cp1252
False
cp1252
mbcs
utf-8

The questions are: How does that happen? And: How can I fix that?

codecs.getwriter([something])(sys.stdout)

seems to be discouraged because it may lead to modules with broken output. Or is it possible to force that to utf-8 in case we did a check for a tty? Better: How to fix that in PyInstaller?

Thanks in advance...

Upvotes: 1

Views: 674

Answers (1)

Bigfoot29
Bigfoot29

Reputation: 70

Thanks to eryksun, the following workaround is working:

STDOUT_ENCODING = str(sys.stdout.encoding)
try:
    PYTHONIOENCODING = str(os.environ["PYTHONIOENCODING"])
except:
    PYTHONIOENCODING = False

# Remark: In case the stdout gets modified, it will only append all information
# that has been written into the pipe until that very moment.
if sys.stdout.isatty() is False:
    print("Program is running in piping mode. (sys.stdout.isatty() is " + str(sys.stdout.isatty()) + ".)")
    if PYTHONIOENCODING is not False:
        print("PYTHONIOENCODING is set to a value. ('" + str(PYTHONIOENCODING) + "')")
        if str(sys.stdout.encoding) != str(PYTHONIOENCODING):
            print("PYTHONIOENCODING is differing from stdout encoding. ('" + str(PYTHONIOENCODING) + "' != '" + STDOUT_ENCODING + "'). This should normally not happen unless the PyInstaller setup is still broken. Setting hard utf-8 workaround.")
            sys.stdout = open(sys.stdout.fileno(), 'w', encoding='utf-8', closefd=False)
            print("PYTHONIOENCODING was differing from stdout encoding. ('" + str(PYTHONIOENCODING) + "' != '" + STDOUT_ENCODING + "'). This should normally not happen unless PyInstaller is still broken. Setting hard utf-8 workaround. New encoding: '" + str(PYTHONIOENCODING) + "'.", "D")
        else:
            print("PYTHONIOENCODING is equal to stdout encoding. ('" + str(PYTHONIOENCODING) + "' == '" + str(sys.stdout.encoding) + "'). - All good.")
    else:
        print("PYTHONIOENCODING is set False. ('" + str(PYTHONIOENCODING) + "'). - Nothing to do.")
else:
    print("Program is running in terminal mode. (sys.stdout.isatty() is " + str(sys.stdout.isatty()) + ".) - All good.")

Trying to set up a new PyInstaller-Environment to see if that fixes it from the start next.

Upvotes: 2

Related Questions