CODEWITHSUNDEEP
pythonbeautifulsoup
PPEGG
PPEGG

Reputation: 3

Python Beautiful Soup HTML to Text

I'm using the BeautifulSoup package to scrape a website.

I extracted the content we are looking for into a variable called l_results by using the following code

l_results = soup.find_all('div',attrs={"class":"gitb-section-content"})

This returns the following data:

[<div class="gitb-section-content" data-section_name="valuable_features">\n<ul>\n<li>Passcode enforcement on devices containing corporate email or data</li>\n<li>The notification of new devices accessing corporate email and VPN connectivity</li>\n<li>Deploying needed applications to device groups</li>\n</ul>\n</div>,
 <div class="gitb-section-content" data-section_name="improvements_to_organization">\n<p>The product has given us complete control of devices allowed to receive company data. It is important that only salaried employees receive corporate email on mobile devices.  Checking and responding to corporate email outside of normal scheduled shifts by hourly employees, can and should be time paid.</p>\n</div>,
 <div class="gitb-section-content" data-section_name="room_for_improvement">\n<p>I would like to see one-click app distribution to a single device or user. Perhaps I need further instruction in this area if it is supposed to function in this way currently. I would also like the ability to add a nagging message to any user that falls out of compliance.</p>\n</div>,
 <div class="gitb-section-content" data-section_name="use_of_solution">\n<p>I've used it for three years.</p>\n</div>,
 <div class="gitb-section-content" data-section_name="stability_issues">\n<p>It does seem that the more devices we added, the slower the management console operates.</p>\n</div>,
 <div class="gitb-section-content" data-section_name="other_advice">\n<p>We are very pleased with the Maas360 product and plan to continue use as our company grows.</p>\n</div>]

Now I am trying to extract the text from the 'p' and 'li' tags, as some reviews may contain both paragraph text as well as list items (wasn't aware of the li originally).
I am able to get results for those ones which do not contain list items by using the following:

for x in l_results:
    review_text += '\n' + ''.join(x.find('p').text)

when the code encounters a review with li in it, I get the following results: 

File "<ipython-input-63-d24fd128d779>", line 2, in <module> 
  review_text += '\n' + ''.join(x.find('p').text)
AttributeError: 'NoneType' object has no attribute 'text'

Upvotes: 0

Views: 139

Answers (1)

OneCricketeer
OneCricketeer

Reputation: 191993

Try getting the paragraphs text only if they exist

for x in l_results:
    review_text += '\n'
    _p = x.find('p')
    if _p:
        review_text += ''.join(_p.text)

Upvotes: 1

Related Questions