CODEWITHSUNDEEP
pythonhtml-parsingbeautifulsoup
Achim
Achim

Reputation: 15702

BeautifulSoup fails to parse long view state

I try to use BeautifulSoup4 to parse the html retrieved from http://exporter.nih.gov/ExPORTER_Catalog.aspx?index=0 If I print out the resulting soup, it ends like this:

kZXI9IjAi"/></form></body></html>

Searching for the last characters 9IjaI in the raw html, I found that it's in the middle of a huge viewstate. BeautifulSoup seems to have a problem with this. Any hint what I might be doing wrong or how to parse such a page?

Upvotes: 0

Views: 707

Answers (1)

Martijn Pieters
Martijn Pieters

Reputation: 1121744

BeautifulSoup uses a pluggable HTML parser to build the 'soup'; you need to try out different parsers, as each will treat a broken page differently.

I had no problems parsing that page with any of the parsers, however:

>>> from beautifulsoup4 import BeautifulSoup
>>> import requests
>>> r = requests.get('http://exporter.nih.gov/ExPORTER_Catalog.aspx?index=0')
>>> for parser in ('html.parser', 'lxml', 'html5lib'):
...     print repr(str(BeautifulSoup(r.text, parser))[-60:])
... 
';\r\npageTracker._trackPageview();\r\n</script>\n</body>\n</html>\n'
'();\r\npageTracker._trackPageview();\r\n</script>\n</body></html>'
'();\npageTracker._trackPageview();\n</script>\n\n\n</body></html>'

Make sure you have the latest BeautifulSoup4 package installed, I have seen consistent problems in the 4.1 series solved in 4.2.

Upvotes: 1

Related Questions