CODEWITHSUNDEEP
pythonfilepython-2.7encodingio
Alain
Alain

Reputation: 27220

"io.open" is improperly opening UTF-16 files

io.open is supposed to be stripping preambles when opening files in various encodings.

For instance, the following file encoded with UTF-8-SIG has the preamble stripped correctly before reading it into a string:

(Note: I'm not opening these files in binary mode. The first line of these logs is to demonstrate the contents of the files that are about to be read.)

# Raw binary, so you can see that it's a proper UTF-8-SIG encoded file
import io; io.open(csv_file_path, 'br').readline()
'\xef\xbb\xbf"EventId","Rate","Attribute1","Attribute2","(\xef\xbd\xa1\xef\xbd\xa5\xcf\x89\xef\xbd\xa5\xef\xbd\xa1)\xef\xbe\x89"\r\n'

# Open file with encoding specified
import io; io.open(csv_file_path, encoding='UTF-8-SIG').readline()
u'"EventId","Rate","Attribute1","Attribute2","(\uff61\uff65\u03c9\uff65\uff61)\uff89"\n'

But while this file with a UTF-16LE encoding is being successfully opened, the preamble is coming with it:

# Raw binary, so you can see that it's a proper UTF-16LE encoded file
import io; io.open(csv_file_path, 'br').readline()
'\xff\xfe"\x00E\x00v\x00e\x00n\x00t\x00I\x00d\x00"\x00,\x00"\x00R\x00a\x00t\x00e\x00"\x00,\x00"\x00A\x00t\x00t\x00r\x00i\x00b\x00u\x00t\x00e\x001\x00"\x00,\x00"\x00A\x00t\x00t\x00r\x00i\x00b\x00u\x00t\x00e\x002\x00"\x00,\x00"\x00(\x00a\xffe\xff\xc9\x03e\xffa\xff)\x00\x89\xff"\x00\r\x00\n'

# Open file with encoding specified
import io; io.open(csv_file_path, encoding='UTF-16LE').readline()
u'\ufeff"EventId","Rate","Attribute1","Attribute2","(\uff61\uff65\u03c9\uff65\uff61)\uff89"\n'

This goes on to break file validation that expects the file contents to start right off with "EventId"...

Am I opening this file incorrectly?

Note that I'm not satisfied having to manually strip out preambles after opening the file - I want to support arbitrary encodings and I expect io.open (with the correct encoding supplied, as determined by chardet) to abstract away the need for me to have a bunch of hard coded preambles to skip if encountered at the beginning of the first line.

Upvotes: 0

Views: 614

Answers (1)

Robᵩ
Robᵩ

Reputation: 168616

According to this answer, you need to use UTF-16, not UTF-16LE.

io.open(csv_file_path, encoding='UTF-16').readline()

Upvotes: 2

Related Questions