CODEWITHSUNDEEP
pythonbeautifulsoup
yung peso
yung peso

Reputation: 1796

removing additional data (html tags) from output?

I scraped a list of stocks and appended the items to a list, but doing so also added extra html elements due to my bs4 query.

Here is my reproducible code:

from bs4 import BeautifulSoup
from urllib.request import Request, urlopen

url = 'https://bullishbears.com/russell-2000-stocks-list/'
hdr = {'User-Agent': 'Mozilla/5.0'}
req = Request(url,headers=hdr)
page = urlopen(req)
soup = BeautifulSoup(page)

divTag = soup.find_all("div", {"class": "thrv_wrapper thrv_text_element"})

stock_list = []
for tag in divTag:
    strongTags = tag.find_all("strong")
    for tag in strongTags:
        for x in tag:      
            stock_list.append(x)

Looking at the outcome of the list, I'm happy with the stock string format followed by a comma after every stock (list of strings). As you can see, I'm also getting other HTML elements that I want removed <br/> and <span>.

stock_list =

[<span data-css="tve-u-17078d9d4a6">RUSSELL 2000 STOCKS LIST</span>,
 <strong><strong><strong><span data-css="tve-u-17031e9c4ac"> We provide you a list of Russell 2000 stocks and companies below</span><span data-css="tve-u-17031e9c4ad">. </span></strong></strong></strong>,
 <strong><strong><span data-css="tve-u-17031e9c4ac"> We provide you a list of Russell 2000 stocks and companies below</span><span data-css="tve-u-17031e9c4ad">. </span></strong></strong>,
 <strong><span data-css="tve-u-17031e9c4ac"> We provide you a list of Russell 2000 stocks and companies below</span><span data-css="tve-u-17031e9c4ad">. </span></strong>,
 <span data-css="tve-u-17031e9c4ac"> We provide you a list of Russell 2000 stocks and companies below</span>,
 <span data-css="tve-u-17031e9c4ad">. </span>,
 'List of Russell 2000 Stocks & Updated Chart',
 'IWM',
 <br/>,
 'SPSM',
 <br/>,
 'VTWO',
 '/RTY',
 <br/>,
 '/M2K',
 'AAN',
 <br/>,
 'AAOI',
 <br/>,
 'AAON',
 <br/>,
 'AAT',
 <br/>,
 'AAWW',
 <br/>,
 'AAXN',
 <br/>,
 'ABCB',
 <br/>,
 'ABEO',
 <br/>,
 'ABG',
 <br/>,
 'ABM',
 <br/>,
 'ABTX',
 <br/>,
 'AC',
 <br/>,
 'ACA',
 <br/>,
 'ACAD',
 <br/>,
 'ACBI',
 <br/>,
 'ACCO',
# More to the list but for brevity I removed the rest.

How can I properly fine tune my bs4 query to only get a list of stocks?

Upvotes: 0

Views: 50

Answers (1)

uingtea
uingtea

Reputation: 6554

you need to split the value because multiple stocks are inside strong tags

<strong>AAN<br>AAOI<br>AAON<br>AAT<br>....</strong>

the code

# better and easier using CSS selector
strongTags = soup.select('.tcb-col .thrv_wrapper.thrv_text_element strong')

stock_list = []
for s in strongTags:
    # .decode_contents() to get innerHTML
    stocks = s.decode_contents().split('<br/>');
    for stock in stocks:
        stock_list.append(stock)

print(stock_list)

Upvotes: 2

Related Questions