CODEWITHSUNDEEP
pythonweb-scrapingbeautifulsoup
jbf
jbf

Reputation: 169

Scraping data with BS4 in Python, nested table

I'm trying to scrape some data from baseball-reference.com. I've written some code to get the data from other parts of the site, where the tables are coded a little more simply, but the particular set of pages is apparently more complicated. Here's the code I have so far.

from urllib.request import urlopen from bs4 import BeautifulSoup

# Declare URL
test_url = 'https://www.baseball-reference.com/boxes/SLN/SLN201704020.shtml'

# Query the website and return the HTML
page = urlopen(test_url)

# Parse the HTML and store
soup = BeautifulSoup(page, 'lxml')

table = soup.find("div", {"class": "table_outer_container"})

This doesn't find the tables that I want though (on this particular page, the two tables with At-Bats, RBIs, HRs, runs, etc.). I've tried a few other things, e.g.

table = soup.find_all("table" , {"class": "sortable stats_table"})

but it doesn't work either. I've also tried to read the site using pandas, with no luck, so if there's an easier way with pandas, I'm open to that too.

Upvotes: 1

Views: 1175

Answers (2)

SIM
SIM

Reputation: 22440

This is another way you can get the data from the two tables:

import requests
from bs4 import BeautifulSoup, Comment

res = requests.get("https://www.baseball-reference.com/boxes/SLN/SLN201704020.shtml",headers={"User-Agent":"Mozilla/5.0"})
soup = BeautifulSoup(res.text, 'lxml')
for comment in soup.find_all(string=lambda text:isinstance(text,Comment)):
    data = BeautifulSoup(comment,"lxml")
    for items in data.select("#ChicagoCubsbatting tr,#StLouisCardinalsbatting tr"):
        tds = ' '.join([' '.join(item.text.split()) for item in items.select("th,td")])
        print(tds)

Partial output:

Batting AB R H RBI BB SO PA BA OBP SLG OPS Pit Str WPA aLI WPA+ WPA- RE24 PO A Details
Kyle Schwarber LF 3 0 2 0 0 1 4 .667 .750 1.000 1.750 20 10 0.170 1.75 0.196 -0.026 1.1 2 0 2B,HBP
Kris Bryant 3B 4 0 0 0 0 3 4 .000 .000 .000 .000 19 13 -0.260 2.31 0.000 -0.260 -1.6 0 0 
Anthony Rizzo 1B 4 0 1 0 0 1 4 .250 .250 .250 .500 14 8 -0.214 2.74 0.035 -0.249 -0.9 5 0

Upvotes: 0

Selçuk
Selçuk

Reputation: 1123

I know this code is complicated or bad. But it gets the job done. You can optimize it over time. 

from bs4 import BeautifulSoup,Comment
import requests

r = requests.get('https://www.baseball-reference.com/boxes/SLN/SLN201704020.shtml')
soup = BeautifulSoup(r.text, 'lxml')

comments = soup.find_all(string=lambda text:isinstance(text, Comment))
for comment in comments:
    comment.extract()

    #After getting rid of comments we need Soup again
    another_soup = BeautifulSoup(str(comment),'lxml')

    tables = another_soup.find_all('table' ,{"class": "sortable stats_table"})

    for table in tables:
        #Since we can't get id from table we are going to use table header as stat type.
        stat_type = ''
        for data in table.find('thead').find_all('tr'):
            stat_type = data.th.text.strip()

        #You only need batting.
        if stat_type != 'Batting': break

        for data in table.find('tbody').find_all('tr'):
            player = data.th.text.strip()
            stats = data.find_all('td')

            stat_ab = stats[0].text
            stat_r = stats[1].text
            stat_h = stats[2].text
            stat_rbi = stats[3].text
            # Table goes on
            print(player,stat_ab,stat_r,stat_h,stat_rbi)

        print('-------------------------------------')

The output is:

Kyle Schwarber LF 3 0 2 0
Kris Bryant 3B 4 0 0 0
Anthony Rizzo 1B 4 0 1 0
Ben Zobrist RF 3 1 0 0
Addison Russell SS 4 0 1 0
Jason Heyward CF 4 1 1 0
Willson Contreras C 4 1 2 3
Jon Lester P 2 0 0 0
Carl Edwards P 0 0 0 0
Koji Uehara P 0 0 0 0
Tommy La Stella PH 1 0 0 0
Pedro Strop P 0 0 0 0
Jon Jay PH 1 0 0 0
Mike Montgomery P 0 0 0 0
Javier Baez 2B 4 0 1 0
-------------------------------------
Dexter Fowler CF 4 1 1 0
Aledmys Diaz SS 5 0 2 0
Matt Carpenter 1B 4 0 1 1
Jhonny Peralta 3B 4 0 1 0
Seung-hwan Oh P 0 0 0 0
Jose Martinez PH 1 1 1 0
Yadier Molina C 3 0 2 0
Stephen Piscotty RF 3 1 1 0
Jedd Gyorko 2B 2 0 0 0
Kolten Wong PH-2B 2 0 0 0
Randal Grichuk LF 4 1 2 3
Carlos Martinez P 3 0 0 0
Greg Garcia 3B 0 0 0 0
-------------------------------------

Upvotes: 1

Related Questions