CODEWITHSUNDEEP
pythonxmlunicodeparsingcodec
screenglow
screenglow

Reputation: 1886

Python: How do I read and parse a unicode utf-8 text file?

I am exporting UTF-8 text from Excel and I want to read and parse the incoming data using Python. I've read all the online info so I've already tried this, for example:

 txtFile = codecs.open( 'halout.txt', 'r', 'utf-8' )
 for line in txtFile:
  print repr( line )

The error I am getting is:

UnicodeDecodeError: 'utf8' codec can't decode byte 0xff in position 0: unexpected code byte

Looking at the text file in a Hex editor, the first values are FFFE I've also tried:

txtFile.seek( 2 )

right after the 'open' but that just causes a different error.

Upvotes: 3

Views: 4411

Answers (4)

Hoang Pham
Hoang Pham

Reputation: 6949

Try to see if the excel file has some blank rows (and then has values again), that might cause the unexpected error.

Upvotes: 0

nosklo
nosklo

Reputation: 223152

That is a BOM

EDIT, from the coments, it seems to be a utf-16 bom

codecs.open('foo.txt', 'r', 'utf-16')

should work.

Upvotes: 2

Mark Byers
Mark Byers

Reputation: 839224

Expanding on Johnathan's comment, this code should read the file correctly:

import codecs
txtFile = codecs.open( 'halout.txt', 'r', 'utf-16' )
for line in txtFile:
   print repr( line )

Upvotes: 2

Jonathan Feinberg
Jonathan Feinberg

Reputation: 45364

That file is not UTF-8; it's UTF-16LE with a byte-order marker.

Upvotes: 5

Related Questions