CODEWITHSUNDEEP
pythonhtmlregexbeautifulsoup
alvas
alvas

Reputation: 122348

How to extract and ignore span in markup? - python

How to extract and ignore span in HTML markup?

My input looks like this:

<ul class="definitions">
<li><span>noun</span> the joining together of businesses which deal with different stages in the production or <a href="sale.html">sale</a> of the same <u slug="product">product</u>, as when a restaurant <a href="chain.html">chain</a> takes over a <a href="wine.html">wine</a> importer</li></ul>

Desired outputs:

label = 'noun' # String embedded between <span>...</span>
meaning = 'the joining together of businesses which deal with different stages in the production or sale of the same product, as when a restaurant chain takes over a wine importer' # the text without the string embedded within <span>...</span>
related_to = ['sale', 'chain', 'wine'] # String embedded between <a>...</a>
utag = ['product'] # String embedded between <u>...</u>

I've tried this:

>>> from bs4 import BeautifulSoup
>>> text = '''<ul class="definitions">
...     <li><span>noun</span> the joining together of businesses which deal with different stages in the production or <a href="sale.html">sale</a> of the same <u slug="product">product</u>, as when a restaurant <a href="chain.html">chain</a> takes over a <a href="wine.html">wine</a> importer</li></ul>'''
>>> bsoup = BeautifulSoup(text)
>>> bsoup.text
u'\nnoun the joining together of businesses which deal with different stages in the production or sale of the same product, as when a restaurant chain takes over a wine importer'

# Getting the `label`
>>> label = bsoup.find('span')
>>> label
<span>noun</span>
>>> label = bsoup.find('span').text
>>> label
u'noun'

# Getting the text.
>>> bsoup.text.strip()
u'noun the joining together of businesses which deal with different stages in the production or sale of the same product, as when a restaurant chain takes over a wine importer'
>>> bsoup.text.strip
>>> definition = bsoup.text.strip() 
>>> definition = definition.partition(' ')[2] if definition.split()[0] == label else definition
>>> definition
u'the joining together of businesses which deal with different stages in the production or sale of the same product, as when a restaurant chain takes over a wine importer'

# Getting the related_to and utag
>>> related_to = [r.text for r in bsoup.find_all('a')]
>>> related_to
[u'sale', u'chain', u'wine']
>>> related_to = [r.text for r in bsoup.find_all('u')]
>>> related_to = [r.text for r in bsoup.find_all('a')]
>>> utag = [r.text for r in bsoup.find_all('u')]
>>> related_to
[u'sale', u'chain', u'wine']
>>> utag
[u'product']

Using BeautifulSoup is okay but it's a little verbose to get what's needed.

Is there any other to achieve the same outputs?

Is there a regex way with some groups to catch the desired outputs?

Upvotes: 4

Views: 340

Answers (2)

siegerts
siegerts

Reputation: 471

PyQuery is another option to using BeautifulSoup. It follows a jQuery like syntax for extracting info out of html.

Also, for regex...something like below can be used.

import re

text = """<ul class="definitions"><li><span>noun</span> the joining together of businesses which deal with different stages in the production or <a href="sale.html">sale</a> of the same <u slug="product">product</u>, as when a restaurant <a href="chain.html">chain</a> takes over a <a href="wine.html">wine</a> importer</li></ul>"""

match_pattern = re.compile(r"""
                (?P<label>(?<=<span>)\w+?(?=</span>)) # create the label \
                                                         item for groupdict()
                 """, re.VERBOSE)

match = match_pattern.search(text)
match.groupdict()

outputs:

{'label': 'noun'}

Using the above as a template, you can build on that with respect to the other html tags too. It uses (?P<name>...) to name the matched pattern (i.e. label) and then a (?=...) lookahead assersion and a positive lookbehind assertion to perform the match.

Also, look in to findall or finditer if you have a doc that has more than once instance of your mentioned text pattern.

Upvotes: 1

alecxe
alecxe

Reputation: 474271

It still has a pretty well-formed structure and you've stated the set of rules clearly. I would still approach it with BeautifulSoup applying the "Extract Method" refactoring method:

from pprint import pprint
from bs4 import BeautifulSoup


data = """
<ul class="definitions">
<li><span>noun</span> the joining together of businesses which deal with different stages in the production or <a href="sale.html">sale</a> of the same <u slug="product">product</u>, as when a restaurant <a href="chain.html">chain</a> takes over a <a href="wine.html">wine</a> importer</li></ul>
"""

def get_info(elm):
    label = elm.find("span")
    return {
        "label": label.text,
        "meaning": "".join(getattr(sibling, "text", sibling) for sibling in label.next_siblings).strip(),
        "related_to": [a.text for a in elm.find_all("a")],
        "utag": [u.text for u in elm.find_all("u")]
    }

soup = BeautifulSoup(data, "html.parser")
pprint(get_info(soup.li))

Prints:

{'label': u'noun',
 'meaning': u'the joining together of businesses which deal with different stages in the production or sale of the same product, as when a restaurant chain takes over a wine importer',
 'related_to': [u'sale', u'chain', u'wine'],
 'utag': [u'product']}

Upvotes: 2

Related Questions