CODEWITHSUNDEEP
pythonutf-8urllib2
Wizzard
Wizzard

Reputation: 12702

Process a page with utf-8

I am just playing with urllib2 and pages with utf-8.

http://www.columbia.edu/~fdc/utf8/

Only getting the first 700 bytes (top segment)

>>> import urllib2
>>> from urllib2 import HTTPError, URLError
>>> import BaseHTTPServer
>>> opener = urllib2.OpenerDirector()
>>> opener.add_handler(urllib2.HTTPHandler())
>>> opener.add_handler(urllib2.HTTPDefaultErrorHandler())
>>> response = opener.open('http://www.columbia.edu/~fdc/utf8/')
>>> content = response.read(700)

Now from here, I would think that the string in the content var would be utf-8 encoded, and should display pretty fine.

however

>>> content
'<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">\n<html>\n<head>\n<BASE href="http://kermit.columbia.edu">\n<META http-equiv="Content-Type" content="text/html; charset=utf-8">\n<title>UTF-8 Sampler</title>\n</head>\n<body bgcolor="#ffffff" text="#000000">\n<h1><tt>UTF-8 SAMPLER</tt></h1>\n\n<big><big>&nbsp;&nbsp;\xc2\xa5&nbsp;\xc2\xb7&nbsp;\xc2\xa3&nbsp;\xc2\xb7&nbsp;\xe2\x82\xac&nbsp;\xc2\xb7&nbsp;$&nbsp;\xc2\xb7&nbsp;\xc2\xa2&nbsp;\xc2\xb7&nbsp;\xe2\x82\xa1&nbsp;\xc2\xb7&nbsp;\xe2\x82\xa2&nbsp;\xc2\xb7&nbsp;\xe2\x82\xa3&nbsp;\xc2\xb7&nbsp;\xe2\x82\xa4&nbsp;\xc2\xb7&nbsp;\xe2\x82\xa5&nbsp;\xc2\xb7&nbsp;\xe2\x82\xa6&nbsp;\xc2\xb7&nbsp;\xe2\x82\xa7&nbsp;\xc2\xb7&nbsp;\xe2\x82\xa8&nbsp;\xc2\xb7&nbsp;\xe2\x82\xa9&nbsp;\xc2\xb7&nbsp;\xe2\x82\xaa&nbsp;\xc2\xb7&nbsp;\xe2\x82\xab&nbsp;\xc2\xb7&nbsp;\xe2\x82\xad&nbsp;\xc2\xb7&nbsp;\xe2\x82\xae&nbsp;\xc2\xb7&nbsp;\xe2\x82\xaf&nbsp;\xc2\xb7&nbsp;&#8377</big></big>\n\n\n\n<p>\n<blockquote>\nFrank da Cruz<br>\n<a hre'

Seems html escaped, so

>>> import HTMLParser
>>> h = HTMLParser.HTMLParser()
>>> h.unescape(content)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/HTMLParser.py", line 390, in unescape
    return re.sub(r"&(#?[xX]?(?:[0-9a-fA-F]+|\w{1,8}));", replaceEntities, s)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/re.py", line 151, in sub
    return _compile(pattern, flags).sub(repl, string, count)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 0: ordinal not in range(128)

So I don't understand. I even tried doing .encode('utf-8') being unescaping, but similar error.

What is the best way to display utf-8 content from a website?

Upvotes: 0

Views: 215

Answers (2)

Martijn Pieters
Martijn Pieters

Reputation: 1121824

You need to decode the page from UTF-8 to Unicode; there are UTF-8 sequences in there (next to non-breaking-space HTML entities):

>>> print h.unescape(content.decode('utf8'))
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<BASE href="http://kermit.columbia.edu">
<META http-equiv="Content-Type" content="text/html; charset=utf-8">
<title>UTF-8 Sampler</title>
</head>
<body bgcolor="#ffffff" text="#000000">
<h1><tt>UTF-8 SAMPLER</tt></h1>

<big><big>  ¥ · £ · € · $ · ¢ · ₡ · ₢ · ₣ · ₤ · ₥ · ₦ · ₧ · ₨ · ₩ · ₪ · ₫ · ₭ · ₮ · ₯ · &#8377</big></big>



<p>
<blockquote>
Frank da Cruz<br>
<a hre

You got encoding and decoding confused; the content is already UTF-8 encoded.

Note that the &#8377 is an error in the page itself, the ; was omitted. A HTML5 parser or browser would probably assume that the ; can be added and decode it anyway:

>>> print h.unescape('&#8377;')
₹

You'd have to fix those entities with a regular expression first:

>>> import re
>>> brokenrefs = re.compile(r'(&#x?[a-e0-9]+)\b', re.I)
>>> print h.unescape(brokenrefs.sub(r'\1;', content.decode('utf8')))
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<BASE href="http://kermit.columbia.edu">
<META http-equiv="Content-Type" content="text/html; charset=utf-8">
<title>UTF-8 Sampler</title>
</head>
<body bgcolor="#ffffff" text="#000000">
<h1><tt>UTF-8 SAMPLER</tt></h1>

<big><big>  ¥ · £ · € · $ · ¢ · ₡ · ₢ · ₣ · ₤ · ₥ · ₦ · ₧ · ₨ · ₩ · ₪ · ₫ · ₭ · ₮ · ₯ · ₹</big></big>



<p>
<blockquote>
Frank da Cruz<br>
<a hre

Upvotes: 3

Daniel Roseman
Daniel Roseman

Reputation: 599590

You've misunderstood your output. There's nothing HTML-encoded there: but when you simply type content into the REPL, it shows you the repr()-ed version of the text.

Doing print content gives you what you expect:

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<BASE href="http://kermit.columbia.edu">
<META http-equiv="Content-Type" content="text/html; charset=utf-8">
<title>UTF-8 Sampler</title>
</head>
<body bgcolor="#ffffff" text="#000000">
<h1><tt>UTF-8 SAMPLER</tt></h1>

<big><big>&nbsp;&nbsp;¥&nbsp;·&nbsp;£&nbsp;·&nbsp;€&nbsp;·&nbsp;$&nbsp;·&nbsp;¢&nbsp;·&nbsp;₡&nbsp;·&nbsp;₢&nbsp;·&nbsp;₣&nbsp;·&nbsp;₤&nbsp;·&nbsp;₥&nbsp;·&nbsp;₦&nbsp;·&nbsp;₧&nbsp;·&nbsp;₨&nbsp;·&nbsp;₩&nbsp;·&nbsp;₪&nbsp;·&nbsp;₫&nbsp;·&nbsp;₭&nbsp;·&nbsp;₮&nbsp;·&nbsp;₯&nbsp;·&nbsp;&#8377</big></big>



<p>
<blockquote>
Frank da Cruz<br>
<a hre

Upvotes: 0

Related Questions