Unicode problems in Python again

Question

I have a strange problem. A friend of mine sent me a text file. If I copy/paste the text and then paste it into my text editor and save it, the following code works. If I choose the option to save the file directly from the browser, the following code breaks. What's going on? Is it the browser's fault for saving invalid characters?

This is an example line.

When I save it, the line says

What�s going on?

When I copy/paste it, the line says

What’s going on?

This is the code:

import codecs

def do_stuff(filename):
    with codecs.open(filename, encoding='utf-8') as f:
        def process_line(line):
            return line.strip()
        lines = f.readlines()
        for line in lines:
           line = process_line(line)
           print line

do_stuff('stuff.txt')

This is the traceback I get:

Traceback (most recent call last):
  File "test-encoding.py", line 13, in <module>
    do_stuff('stuff.txt')
  File "test-encoding.py", line 8, in do_stuff
    lines = f.readlines()
  File "/home/somebody/.python/lib64/python2.7/codecs.py", line 679, in readlines
    return self.reader.readlines(sizehint)
  File "/home/somebody/.python/lib64/python2.7/codecs.py", line 588, in readlines
    data = self.read()
  File "/home/somebody/.python/lib64/python2.7/codecs.py", line 477, in read
    newchars, decodedbytes = self.decode(data, self.errors)
UnicodeDecodeError: 'utf8' codec can't decode byte 0x92 in position 4: invalid start byte

What can I do in such cases?

How can I distribute the script if I don't know what encoding the user who runs it will use?

Fixed:

codecs.open(filename, encoding='utf-8', errors='ignore') as f:

Answer 1

The "file-oriented" part of the browser works with raw bytes, not characters. The specific encoding used by the page should be specified either in the HTTP headers or in the HTML itself. You must use this encoding instead of assuming that you have UTF-8 data.