CODEWITHSUNDEEP
pythonhtmlwebbeautifulsoupscreen-scraping
CS498
CS498

Reputation: 3

python web scraping beautiful soup and adding to a list

I am trying to learn to web scrape using Python and BeautifulSoup. My problem is when attempting to add "scraped" items to a new list, only the final entry in the relevant tags is displaying when I print the lists. How do I add each combination as a list item?

import requests

    standings = requests.get('http://games.espn.com/ffl/tools/finalstandings?leagueId=379978&seasonId=2012')


from bs4 import BeautifulSoup

soup = BeautifulSoup(standings.text, 'html.parser')

## Ask BeautifulSoup to find all of the records

pat = soup.find_all('tr', attrs={'class':'sortableRow evenRow'})
teams = []
for x in pat:
        name1 = x.find('a').text
        record1 = x.find('td', {'class':'sortableREC'}).text
        pf1 = x.find('td', {'class':'sortablePF'}).text
        pa1 = x.find('td', {'class':'sortablePA'}).text
        pfg1 = x.find('td', {'class':'sortablePFG'}).text
        pag1 = x.find('td', {'class':'sortablePAG'}).text
        diff1 = x.find('td', {'class':'sortableDIFF'}).text


 teams.append((name1, record1, pf1, pa1, pfg1, pag1, diff1))

odd =soup.find_all('tr', attrs={'class':'sortableRow oddRow'})

teams2 = []
for team in odd:
        name2 = team.find('a').text
        record2 = team.find('td', {'class':'sortableREC'}).text
        pf2 = team.find('td', {'class':'sortablePF'}).text
        pa2 = team.find('td', {'class':'sortablePA'}).text
        pfg2 = team.find('td', {'class':'sortablePFG'}).text
        pag2 = team.find('td', {'class':'sortablePAG'}).text
        diff2 = team.find('td', {'class':'sortableDIFF'}).text
teams2.append((name2, record2, pf2, pa2, pfg2, pag2, diff2))

Upvotes: 0

Views: 323

Answers (1)

kungphu
kungphu

Reputation: 4859

Assuming this isn't just a code formatting mistake, it's because your .append(...) calls aren't inside of your loops. Indent them to the same level as your variable settings (which aren't necessary if you just need those values during list creation) and you should get all relevant values.

Upvotes: 0

Related Questions