CODEWITHSUNDEEP
pythonparsingxml-parsingbeautifulsoupsgml
scip
scip

Reputation: 153

Python: Parsing SGML

I'm trying to parse some SGML like the following in Python:

<!DOCTYPE lewis SYSTEM "lewis.dtd">
<TEXT>
    <TITLE>One</TITLE>
    <BODY>Sample One</BODY>
</TEXT>
<TEXT>
    <TITLE>Two</TITLE>
    <BODY>Sample Two</BODY>
</TEXT>

Here, I'm just looking for everything inside the <BODY> tags (i.e. ["Sample One", "Sample Two"]).

I've tried using BeautifulSoup, but it doesn't like the <!DOCTYPE> in the first line and also expects everything to be wrapped around a root tag like <everything></everything>. While I can manually make these changes before passing it into BeautifulSoup, it feels a bit too hacky.

I'm pretty new to SGML, and also not married to BeautifulSoup, so I'm open to any suggestions.

(For those curious: my specific usecase is the reuters21578 dataset.)

Upvotes: 3

Views: 4222

Answers (1)

Anand S Kumar
Anand S Kumar

Reputation: 91007

You can try using 'html.parser' as the parser instead of lxml-xml. lxml-xml would expect the text to be correct xml , which is not the case.

Example/Demo -

>>> from bs4 import BeautifulSoup
>>> s = """<!DOCTYPE lewis SYSTEM "lewis.dtd">
... <TEXT>
...     <TITLE>One</TITLE>
...     <BODY>Sample One</BODY>
... </TEXT>
... <TEXT>
...     <TITLE>Two</TITLE>
...     <BODY>Sample Two</BODY>
... </TEXT>"""
>>> soup = BeautifulSoup(s,'html.parser')
>>> soup.find_all('body')
[<body>Sample One</body>, <body>Sample Two</body>]

Upvotes: 5

Related Questions