CODEWITHSUNDEEP
pythonbeautifulsoup
n1c9
n1c9

Reputation: 2687

Iterate through an entire table with BeautifulSoup?

Trying to scrape all of the player names and fantasy info on the players listed on this site. I can find the table absolutely fine, but the trouble starts when I try and iterate over the entire table. Here's the code I've written so far:

from bs4 import BeautifulSoup
from urllib.request import urlopen

nfl = 'http://www.fantasypros.com/nfl/adp/overall.php'
html = urlopen(nfl)
soup = BeautifulSoup(html.read(), "lxml")

table = soup.find('tbody').find_next('tbody')

playername = table.find('td').find_next('td')
for row in table:
    print(playername)

Expected output:

Adrian Peterson MIN, 5
Le'Veon Bell PIT, 11

and so on and so forth for the rest of the players on the chart.

Actual output:

Adrian Peterson MIN, 5 
Adrian Peterson MIN, 5 
Adrian Peterson MIN, 5

and so on for over 400 iterations.

Where is my for loop going wrong?

Upvotes: 1

Views: 2791

Answers (1)

alecxe
alecxe

Reputation: 473763

You need to make the search in the context of a particular table:

for row in table:
    print(row.find('td').find_next('td'))

Though, I would approach the problem differently. The desired table has an id:

table = soup.find('table', id="data")
for row in table.find_all("tr")[1:]:  # skipping header row
    cells = row.find_all("td")
    print(cells[0].text, cells[1].find('a').text)

Prints:

(u'1', u'Adrian Peterson')
(u'2', u"Le'Veon Bell")
(u'3', u'Eddie Lacy')
(u'4', u'Jamaal Charles')
(u'5', u'Marshawn Lynch')
...

Upvotes: 1

Related Questions