CODEWITHSUNDEEP
python-3.xbeautifulsoup
phlprcks
phlprcks

Reputation: 65

Extracting contents of nested <p> Tags with Beautiful Soup

I am having a hard time using the advantages of beautiful soup for my use case. There are many similar but not always equal nested p tags where I want to get the contents from. Examples as follows:

<p><span class="example" data-location="1:20">20</span>normal string</p>
<p><span class="example" data-location="1:21">21</span>this text <strong>belongs together</strong></p>
<p><span class="example" data-location="1:22">22</span>some text (<span class="referencequote">a reference text</span>)that might continue</p>
<p><span class="example" data-location="1:23">23</span>more text</p><div class="linebreak"></div>
<p><span class="example" data-location="1:22">24</span>text with (<span class="referencequote">first</span>)two references <span class="referencequote">first</span>.</p>

I need to save the string of the span tag as well as the strings inside the p tag, no matter its styling and if applicable the referencequote. So from examples above I would like to extract:

example = 20, text = 'normal string', reference = []
example = 21, text = 'this text belongs together', reference = []
example = 22, text = 'some text that might continue', reference = ['a reference text']
example = 23, text = 'more text', reference = []
example = 24, text = 'text with two references', reference = ['first', 'second']

What I was trying is to collect all items with the "example" class and then looping though its parents contents.

for span in bs.find_all("span", {"class": "example"}):
    references = []
        for item in span.parent.contents:
            if (type(item) == NavigableString):
                text= item
            elif (item['class'][0]) == 'verse':
                number= int(item.string)
            elif (item['class']) == 'referencequote':
                references.append(item.string)
            else:
                #how to handle <strong> tags?
        verses.append(MyClassObject(n=number, t=text, r=references))

My approach is very prone to error and there might be even more tags like <strong>, <em> that I am ignoring right now. The get_text() method unfortunately gives back sth like '22 some text a reference text that might continue'.

There must be an elegant way to extract this information. Could you give me some ideas for other approaches? Thanks in advance!

Upvotes: 0

Views: 278

Answers (2)

phlprcks
phlprcks

Reputation: 65

A different approach I found out - without regex and maybe more robust to different spans that might come up

    for s in bsItem.select('span'):
        if s['class'][0] == 'example' :
            # do whatever needed with the content of this span 
            s.extract()
        elif s['class'][0] == 'referencequote':
            # do whatever needed with the content of this span 
            s.extract()
       # check for all spans with a class where you want the text excluded

    # finally get all the text
    text = span.parent.text.replace(' ()', '')

maybe that approach is of interest for someone reading this :)

Upvotes: 0

dabingsou
dabingsou

Reputation: 2469

Try this.

from simplified_scrapy.core.regex_helper import replaceReg
from simplified_scrapy import SimplifiedDoc,utils
html = '''
<p><span class="example" data-location="1:20">20</span>normal string</p>
<p><span class="example" data-location="1:21">21</span>this text <strong>belongs together</strong></p>
<p><span class="example" data-location="1:22">22</span>some text (<span class="referencequote">a reference text</span>)that might continue</p>
<p><span class="example" data-location="1:23">23</span>more text</p><div class="linebreak"></div>
<p><span class="example" data-location="1:22">24</span>text with (<span class="referencequote">first</span>)two references <span class="referencequote">second</span>.</p>
'''
html = replaceReg(html,"<[/]*strong>","") # Pretreatment
doc = SimplifiedDoc(html)
ps = doc.ps
for p in ps:
    text = ''.join(p.spans.nextText())
    text = replaceReg(text,"[()]+","") # Remove ()
    span = p.span # Get first span
    spans = span.getNexts(tag="span").text # Get references
    print (span["class"], span.text, text, spans)

Result:

example 20 normal string []
example 21 this text belongs together []
example 22 some text that might continue ['a reference text']
example 23 more text []
example 24 text with two references. ['first', 'second']

Here are more examples. https://github.com/yiyedata/simplified-scrapy-demo/tree/master/doc_examples

Upvotes: 1

Related Questions