CODEWITHSUNDEEP
pythonhtmlbeautifulsoup
Michael C
Michael C

Reputation: 13

Python BeautifulSoup cannot find table ID

I am running into some trouble scraping a table using BeautifulSoup. Here is my code

from urllib.request import urlopen
from bs4 import BeautifulSoup

site = "http://www.sports-reference.com/cbb/schools/clemson/2014.html"
page = urlopen(site)
soup = BeautifulSoup(page,"html.parser")

stats = soup.find('table',  id = 'totals')

In [78]: print(stats)
None

When I right click on the table to inspect the element the HTML looks as I'd expect, however when I view the source the only element with id = 'totals' is commented out. Is there a way to scrape a table from the commented source code?

I have referenced this post but can't seem to replicate their solution.

Here is a link to the webpage I am interested in. I'd like to scrape the table labeled "Totals" and store it as a data frame.

I am relatively new to Python, HTML, and web scraping. Any help would be greatly appreciated.

Thanks in advance.

Michael

Upvotes: 1

Views: 2182

Answers (2)

Costis94
Costis94

Reputation: 250

You can do this:

from urllib2 import *
from bs4 import BeautifulSoup

site = "http://www.sports-reference.com/cbb/schools/clemson/2014.html"
page = urlopen(site)
soup = BeautifulSoup(page,"lxml")

stats = soup.findAll('div', id = 'all_totals')
print stats

Please inform me if I helped!

Upvotes: 0

clockwatcher
clockwatcher

Reputation: 3373

Comments are string instances in BeautifulSoup. You can use BeautifulSoup's find method with a regular expression to find the particular string that you're after. Once you have the string, have BeautifulSoup parse that and there you go.

In other words,

import re
from urllib.request import urlopen
from bs4 import BeautifulSoup

site = "http://www.sports-reference.com/cbb/schools/clemson/2014.html"
page = urlopen(site)
soup = BeautifulSoup(page,"html.parser")

stats_html = soup.find(string=re.compile('id="totals"'))
stats_soup = BeautifulSoup(stats_html, "html.parser")

print(stats_soup.table.caption.text)

Upvotes: 1

Related Questions