CODEWITHSUNDEEP
pythonpython-2.7beautifulsoup
Ravash Jalil
Ravash Jalil

Reputation: 820

Scraping Table With Python/BS4

Im trying to scrape the "Team Stats" table from http://www.pro-football-reference.com/boxscores/201602070den.htm with BS4 and Python 2.7. However Im unable to get anywhere close to it, 

url = 'http://www.pro-football-reference.com/boxscores/201602070den.htm'
page = requests.get(url)
soup = BeautifulSoup(page.text, "html5lib")
table=soup.findAll('table', {'id':"team_stats", "class":"stats_table"})  
print table

I thought something like the above code would work but no luck.

Upvotes: 1

Views: 2439

Answers (1)

alecxe
alecxe

Reputation: 473903

The problem in this case is that the "Team Stats" table is located inside a comment in the HTML source which you download with requests. Locate the comment and reparse it with BeautifulSoup into a "soup" object:

import requests
from bs4 import BeautifulSoup, NavigableString

url = 'http://www.pro-football-reference.com/boxscores/201602070den.htm'
page = requests.get(url, headers={'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36'})

soup = BeautifulSoup(page.content, "html5lib")
comment = soup.find(text=lambda x: isinstance(x, NavigableString) and "team_stats" in x)

soup = BeautifulSoup(comment, "html5lib")
table = soup.find("table", id="team_stats")
print(table)

And/or, you can load the table into, for example, a pandas dataframe which is very convenient to work with:

import pandas as pd
import requests
from bs4 import BeautifulSoup
from bs4 import NavigableString

url = 'http://www.pro-football-reference.com/boxscores/201602070den.htm'
page = requests.get(url, headers={'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36'})

soup = BeautifulSoup(page.content, "html5lib")
comment = soup.find(text=lambda x: isinstance(x, NavigableString) and "team_stats" in x)

df = pd.read_html(comment)[0]
print(df)

Prints:

            Unnamed: 0            DEN            CAR
0          First Downs             11             21
1         Rush-Yds-TDs        28-90-1       27-118-1
2    Cmp-Att-Yd-TD-INT  13-23-141-0-1  18-41-265-0-1
3         Sacked-Yards           5-37           7-68
4       Net Pass Yards            104            197
5          Total Yards            194            315
6         Fumbles-Lost            3-1            4-3
7            Turnovers              2              4
8      Penalties-Yards           6-51         12-102
9     Third Down Conv.           1-14           3-15
10   Fourth Down Conv.            0-0            0-0
11  Time of Possession          27:13          32:47

Upvotes: 2

Related Questions