CODEWITHSUNDEEP
pythonhtmlbeautifulsoup
TankorSmash
TankorSmash

Reputation: 12747

Parsing a specific website crashes the Python process

Looking to parse an HTML page for images (from http://www.z-img.com), and when I load the page into BeautifulSoup (bs4), Python crashes. The "problem details" shows that etree.pyd was the "Fault Module Name", which means its probably a parsing error, but so far, I can't quite nail down the cause of it.

Here's the simplest code I can boil it down to, on Python2.7:

import requests, bs4

url = r"http://z-img.com/search.php?&ssg=off&size=large&q=test"
r = requests.get(url)
html = r.content
#or 
#import urllib2
#html = urllib2.urlopen(url).read()
soup  = bs4.BeautifulSoup(html)

along with a sample output on PasteBin (http://pastebin.com/XYT9g4Lb), after I had passed it through JsBeautifier.com.

Upvotes: 0

Views: 531

Answers (2)

icktoofay
icktoofay

Reputation: 129019

This is a bug that was fixed in lxml version 2.3.5. Upgrade to version 2.3.5 or later.

Upvotes: 1

TankorSmash
TankorSmash

Reputation: 12747

Oh, there you go, naturally the first thing I try after I submit the question is the solution: the <!DOCTYPE> tag seems to be at the root of it. I created a new HTML file, temp.html:

<!DOCTYPE>
<html>
</html>

and passed that to BeautifulSoup as an HTML string, and that was enough to crash Python again. So I just need to remove that tag before I pass the HTML to BeautifulSoup in the future:

import requests, bs4

url = r"http://z-img.com/search.php?&ssg=off&size=large&q=test"
r = requests.get(url)
html = r.content
#or 
#import urllib2
#html = urllib2.urlopen(url).read()

#replace the declaration with nothing, and my problems are solved
html = html.replace(r"<!DOCTYPE>", "")
soup  = bs4.BeautifulSoup(html)

Hope this saves someone else some time.

Upvotes: 0

Related Questions