CODEWITHSUNDEEP
pythonpython-2.7web-scrapingbeautifulsouphtml-parsing
herkyonparade
herkyonparade

Reputation: 179

Scraping data with BeautifulSoup issue

newby web scraper here. I am wondering why I can't extract the information I'm needing. The code as follows: 

from bs4 import BeautifulSoup
import requests

url = "http://www.mortgagenewsdaily.com/mortgage_rates/"

r = requests.get(url)

soup = BeautifulSoup(r.content)

table = soup.find("table", {"class": "rangetable mtg-rates"})

for trs in table.find_all("tr", {"class": "rate-row"}):
    for tds in trs.find_all("td"):
        try:
            product = tds[0].get_text()
            today = tds[1].get_text()
            yesterday = tds[2].get_text()
            change = tds[3].get_text()
            low = tds[4].get_text()
            high = tds[5].get_text()
        except:
            print "-"
            continue

Do I need to pay attention to the classes within the tds? And, is there a better/simpler way to scrape this information?

Thanks for your help!

Upvotes: 1

Views: 147

Answers (1)

alecxe
alecxe

Reputation: 474161

Your main problem is that you have an extra loop over tds.

Also, you can apply the following improvements to the code:

  • use CSS Selectors to get to the appropriate tr elements
  • instead of having a variable for each cell, use a data structure, for example, namedtuple

The improved working code:

from collections import namedtuple
from bs4 import BeautifulSoup
import requests

url = "http://www.mortgagenewsdaily.com/mortgage_rates/"
r = requests.get(url)

soup = BeautifulSoup(r.content)

Item = namedtuple('Item', "product,today,yesterday,change,low,high")
for tr in soup.select("table.mtg-rates tr.rate-row"):
    item = Item(*(td.get_text(strip=True) for td in tr.find_all("td")))
    print item

Prints:

Item(product=u'30 Yr FRM', today=u'3.62%', yesterday=u'3.64%', change=u'-0.023.62%', low=u'3.60%', high=u'4.56%')
Item(product=u'15 Yr FRM', today=u'3.00%', yesterday=u'3.02%', change=u'-0.023.00%', low=u'2.98%', high=u'3.55%')
Item(product=u'FHA 30 Year Fixed', today=u'3.25%', yesterday=u'3.25%', change=u'--3.25%', low=u'3.25%', high=u'4.25%')
Item(product=u'Jumbo 30 Year Fixed', today=u'3.61%', yesterday=u'3.63%', change=u'-0.023.61%', low=u'3.58%', high=u'4.38%')
Item(product=u'5/1 Yr ARM', today=u'3.23%', yesterday=u'3.22%', change=u'+0.013.23%', low=u'3.17%', high=u'3.26%')

Or, in case of a dict:

headers = ['product', 'today', 'yesterday' ,' change', 'low', 'high']
for tr in soup.select("table.mtg-rates tr.rate-row"):
    item = dict(zip(headers, [td.get_text(strip=True) for td in tr.find_all("td")]))
    print item

Prints:

{'product': u'30 Yr FRM', 'yesterday': u'3.64%', 'high': u'4.56%', 'low': u'3.60%', 'today': u'3.62%', ' change': u'-0.023.62%'}
{'product': u'15 Yr FRM', 'yesterday': u'3.02%', 'high': u'3.55%', 'low': u'2.98%', 'today': u'3.00%', ' change': u'-0.023.00%'}
{'product': u'FHA 30 Year Fixed', 'yesterday': u'3.25%', 'high': u'4.25%', 'low': u'3.25%', 'today': u'3.25%', ' change': u'--3.25%'}
{'product': u'Jumbo 30 Year Fixed', 'yesterday': u'3.63%', 'high': u'4.38%', 'low': u'3.58%', 'today': u'3.61%', ' change': u'-0.023.61%'}
{'product': u'5/1 Yr ARM', 'yesterday': u'3.22%', 'high': u'3.26%', 'low': u'3.17%', 'today': u'3.23%', ' change': u'+0.013.23%'}

Upvotes: 1

Related Questions