CODEWITHSUNDEEP
python-3.xbeautifulsoup
fuzzyworm
fuzzyworm

Reputation: 43

How to handle td data without tag in beautifulsoup?

Here is some data:

<tr>
<td data-th='Stock Symbol'>
<a href="/dividend-stocks/technology/personal-computers/aapl-apple-inc/">AAPL</a>
</td>
<td data-th='Company Name'>
<a href="/dividend-stocks/technology/personal-computers/aapl-apple-inc/">Apple Inc.<div class='label label-primary' style='margin-left: 5px;'>
</div></a>
</td>
<td data-th='DARS™ Rating'>
<a href='/premium/signup.php' class='restricted' style='vertical-align: middle;'></a>
</td>
<td data-th='Dividend Yield'>
1.65%
</td>
<td data-th='Closing Price'>
$153.18
</td>
<td data-th='Annualized Dividend'>
$2.52
</td>
<td data-th='Ex-Div Date'>
<span style="white-space: nowrap;">2017-05-11</span>
</td>
<td data-th='Pay Date'>
<span style="white-space: nowrap;">2017-05-18</span>
</td>
</tr>

I need to get the values of 1.65%, $153.18 and $2.52. They are all by themselves on one line without tag.

This code returns nothing. How can I get around this? Thanks.

import requests
from bs4 import BeautifulSoup
url = "http://www.dividend.com/dividend-stocks/dow-30-dividend-stocks.php"
r = requests.get(url)
soup = BeautifulSoup(r.content, "html.parser")

for tds in soup.find_all("td"):
   print(tds)

Upvotes: 1

Views: 599

Answers (1)

wpedrak
wpedrak

Reputation: 687

I find out that html.parser is not the best choice in this case. Let's try html5lib instead. Type (linux)

sudo apt-get install python-html5lib

to install new parser. Link to BF+html5lib docs.

This is working code (for printing text of mentioned tds):

import requests
from bs4 import BeautifulSoup
url = "http://www.dividend.com/dividend-stocks/dow-30-dividend-stocks.php"
r = requests.get(url)
soup = BeautifulSoup(r.content, "html5lib")
interesting_tds = ['Dividend Yield', 'Closing Price', 'Annualized Dividend']

for td in soup.find_all("td"):
    if td.get('data-th') in interesting_tds:
        print(td.text.strip())
        # or just process td object

Upvotes: 2

Related Questions