CODEWITHSUNDEEP
pythonhtmlpython-2.7beautifulsoup
TextGeek
TextGeek

Reputation: 1247

BeautifulSoup (bs4) parsing wrong

Parsing this sample document with bs4, from python 2.7.6:

<html>
<body>
<p>HTML allows omitting P end-tags.

<p>Like that and this.

<p>And this, too.

<p>What happened?</p>

<p>And can we <p>nest a paragraph, too?</p></p>

</body>
</html>

Using:

from bs4 import BeautifulSoup as BS
...
tree = BS(fh)

HTML has, for ages, allowed omitted end-tags for various element types, including P (check the schema, or a parser). However, bs4's prettify() on this document shows that it doesn't end any of those paragraphs until it sees </body>:

<html>
 <body>
  <p>
   HTML allows omitting P end-tags.
   <p>
    Like that and this.
    <p>
     And this, too.
     <p>
      What happened?
     </p>
     <p>
      And can we
      <p>
       nest a paragraph, too?
      </p>
     </p>
    </p>
   </p>
  </p>
 </body>

It's not prettify()'s fault, because traversing the tree manually I get the same structure:

<[document]>
    <html>
        ␊
        <body>
            ␊
            <p>
                HTML allows omitting P end-tags.␊␊
                <p>
                    Like that and this.␊␊
                    <p>
                        And this, too.␊␊
                        <p>
                            What happened?
                        </p>
                        ␊
                        <p>
                            And can we 
                            <p>
                                nest a paragraph, too?
                            </p>
                        </p>
                        ␊
                    </p>
                </p>
            </p>
        </body>
        ␊
    </html>
    ␊
</[document]>

Now, this would be the right result for XML (at least up to </body>, at which point it should report a WF error). But this ain't XML. What gives?

Upvotes: 3

Views: 2924

Answers (1)

TextGeek
TextGeek

Reputation: 1247

The doc at http://www.crummy.com/software/BeautifulSoup/bs4/doc/#installing-a-parser tells how to get BS4 to use different parsers. Apparently the default is html.parse, which the BS4 doc says is broken before Python 2.7.3, but apparently still has the problem described above in 2.7.6.

Switching to "lxml" was unsuccessful for me, but switching to "html5lib" produces the correct result:

tree = BS(htmSource, "html5lib")

Upvotes: 3

Related Questions