CODEWITHSUNDEEP
pythonhtmlbeautifulsoup
ks123321
ks123321

Reputation: 61

BeautifulSoup - stripping out non-breaking spaces from HTML

I am trying to scrape a number of 10K Risk Factor sections, e.g. https://www.sec.gov/Archives/edgar/data/1321502/000143774910004615/andain_10k-123106.htm

One of my problems is that I'm trying to match a few strings exactly (such as "Risk Factors") but sometimes there are several non-breaking spaces between the words

I am hoping I can just strip them out as they're not useful to me, so I tried:

url = 'https://www.sec.gov/Archives/edgar/data/1321502/000143774910004615/andain_10k-123106.htm'
page = requests.get(url)
soup = BeautifulSoup(page.text.replace("\xa0"," "), 'html.parser')

And then searching the soup (in the usual way) to test whether it worked:

soup.find_all(string="ITEM 1A.\xa0\xa0RISK FACTORS")

But the output still contains non-breaking spaces, which it shouldn't:

Out[42]: ['ITEM 1A.\xa0\xa0RISK FACTORS']

What am I doing wrong?

Upvotes: 1

Views: 457

Answers (1)

baduker
baduker

Reputation: 20042

Try this:

import requests
from bs4 import BeautifulSoup

url = 'https://www.sec.gov/Archives/edgar/data/1321502/000143774910004615/andain_10k-123106.htm'
page = requests.get(url)
soup = BeautifulSoup(page.content, 'html.parser')
cleaned_up = [
    i.getText(strip=True).replace(u"\xa0", " ")
    for i in soup.find_all("font") if i.getText().startswith("ITEM")
]
print(cleaned_up[1])

Output:

ITEM 1A.  RISK FACTORS

Upvotes: 1

Related Questions