Use html to go into url and scrape tables

Question

I need to scrape tables from individual pages of players, but sometimes the search will go to a list of players if there are multiple with the same name. I want the one that played in the NBA. For example, for Sergio Rodriguez, a list shows up (https://basketball.realgm.com/search?q=Sergio+Rodriguez), so instead of going to the individual page, it spits out "No international table for Sergio Rodriguez." I want to go into the individual page of the Sergio Rodriguez that played in the NBA, who is second in the list, and scrape tables, but I'm not sure how to go about it. How do i use the html to go into that specific player's page and continue the scraping?

The HTML:



Sergio Rodriguez Febles
SF
6-7
202
Oct 18, 1993
2015
-
-


Sergio Rodriguez
PG
6-3
176
Jun 12, 1986
2006
-
NYK, PHL, POR, SAC


Sergio Rodriguez
SG
6-4
-
Oct 12, 1977
1999
-
-

import requests
from bs4 import BeautifulSoup
import pandas as pd


playernames=['Carlos Delfino', 'Sergio Rodriguez']

result = pd.DataFrame()
for name in playernames:

    fname=name.split(" ")[0]
    lname=name.split(" ")[1]
    url="https://basketball.realgm.com/search?q={}+{}".format(fname,lname)
    response = requests.get(url)

    soup = BeautifulSoup(response.content, 'html.parser')

    # check the response url
    if (response.url == "https://basketball.realgm.com/search..."):
        # parse the search results, finding players who played in NBA
        ... get urls from the table ...
        soup.table...  # etc.
        foreach url in table:
            response = requests.get(player_url)
            soup = BeautifulSoup(response.content, 'html.parser')
            # call the parse function for a player page
            ...
            parse_player(soup)
    else: # we have a player page
        # call the parse function for a player page, same as above
        ...
        parse_player(soup)

    try:
        table1 = soup.find('h2',text='International Regular Season Stats - Per Game').findNext('table')
        table2 = soup.find('h2',text='International Regular Season Stats - Advanced Stats').findNext('table')

        df1 = pd.read_html(str(table1))[0]
        df2 = pd.read_html(str(table2))[0]

        commonCols = list(set(df1.columns) & set(df2.columns))
        df = df1.merge(df2, how='left', on=commonCols)
        df['Player'] = name

    except:
        print ('No international table for %s.' %name)
        df = pd.DataFrame([name], columns=['Player'])

KunduK · Accepted Answer

Use if condition check if the text of the element matches with Sergio Rodriguez then go to that block and get the latest url and then get the soup ans so on..

import requests
from bs4 import BeautifulSoup
import pandas as pd


playernames=['Carlos Delfino', 'Sergio Rodriguez','Nikola Jokic','Brandon Jennings','Thon Maker']

result = pd.DataFrame()
for name in playernames:

    fname=name.split(" ")[0]
    lname=name.split(" ")[1]
    url="https://basketball.realgm.com/search?q={}+{}".format(fname,lname)
    response = requests.get(url)
    soup = BeautifulSoup(response.content, 'html.parser')

    #Add check here if matches with `Sergio Rodriguez` then go to if clause
    if soup.find('a',text=name).text==name:
        url="https://basketball.realgm.com"+soup.find('a',text=name)['href']
        print(url)
        response = requests.get(url)
        soup = BeautifulSoup(response.text, 'lxml')

    try:
        table1 = soup.find('h2',text='International Regular Season Stats - Per Game').findNext('table')
        table2 = soup.find('h2',text='International Regular Season Stats - Advanced Stats').findNext('table')

        df1 = pd.read_html(str(table1))[0]
        df2 = pd.read_html(str(table2))[0]

        commonCols = list(set(df1.columns) & set(df2.columns))
        df = df1.merge(df2, how='left', on=commonCols)
        df['Player'] = name
        print(df)
    except:
        print ('No international table for %s.' %name)
        df = pd.DataFrame([name], columns=['Player'])

Console you can see df is printing like.

  https://basketball.realgm.com/player/Carlos-Delfino/Summary/446
       Season                               Team  ...    PER          Player
0     2002-03         Fortituto Kontatto Bologna  ...  15.38  Carlos Delfino
1   2003-04 *                          All Teams  ...  16.08  Carlos Delfino
2   2003-04 *         Fortituto Kontatto Bologna  ...  15.15  Carlos Delfino
3   2003-04 *         Fortituto Kontatto Bologna  ...  16.58  Carlos Delfino
4     2008-09                             Khimki  ...  19.48  Carlos Delfino
5     2016-17                       Boca Juniors  ...  14.82  Carlos Delfino
6   2017-18 *                          All Teams  ...  -0.76  Carlos Delfino
7   2017-18 *  KIROLBET Baskonia Vitoria-Gasteiz  ...   0.21  Carlos Delfino
8   2017-18 *  KIROLBET Baskonia Vitoria-Gasteiz  ...  -6.67  Carlos Delfino
9   2018-19 *                          All Teams  ...  15.37  Carlos Delfino
10  2018-19 *                        Fiat Torino  ...  13.67  Carlos Delfino
11  2018-19 *         Fortituto Kontatto Bologna  ...  22.76  Carlos Delfino
12  2018-19 *                        Fiat Torino  ...  12.78  Carlos Delfino

[13 rows x 41 columns]
https://basketball.realgm.com/player/Sergio-Rodriguez/Summary/85
       Season                       Team  ...    PER            Player
0     2003-04       Movistar Estudiantes  ...   0.00  Sergio Rodriguez
1   2004-05 *                  All Teams  ...  13.81  Sergio Rodriguez
2   2004-05 *       Movistar Estudiantes  ...  14.55  Sergio Rodriguez
3   2004-05 *       Movistar Estudiantes  ...  11.17  Sergio Rodriguez
4   2005-06 *                  All Teams  ...  12.50  Sergio Rodriguez
5   2005-06 *       Movistar Estudiantes  ...  13.80  Sergio Rodriguez
6   2005-06 *       Movistar Estudiantes  ...   6.09  Sergio Rodriguez
7   2010-11 *                  All Teams  ...  13.45  Sergio Rodriguez
8   2010-11 *                Real Madrid  ...  15.26  Sergio Rodriguez
9   2010-11 *                Real Madrid  ...  12.83  Sergio Rodriguez
10  2011-12 *                  All Teams  ...  14.07  Sergio Rodriguez
11  2011-12 *                Real Madrid  ...  14.84  Sergio Rodriguez
12  2011-12 *                Real Madrid  ...   0.00  Sergio Rodriguez
13  2011-12 *                Real Madrid  ...  15.55  Sergio Rodriguez
14  2012-13 *                  All Teams  ...  17.57  Sergio Rodriguez
15  2012-13 *                Real Madrid  ...  19.51  Sergio Rodriguez
16  2012-13 *                Real Madrid  ...   0.00  Sergio Rodriguez
17  2012-13 *                Real Madrid  ...  30.02  Sergio Rodriguez
18  2013-14 *                  All Teams  ...  22.89  Sergio Rodriguez
19  2013-14 *                Real Madrid  ...  22.05  Sergio Rodriguez
20  2013-14 *                Real Madrid  ...  27.16  Sergio Rodriguez
21  2013-14 *                Real Madrid  ...  18.64  Sergio Rodriguez
22  2014-15 *                  All Teams  ...  19.18  Sergio Rodriguez
23  2014-15 *                Real Madrid  ...  18.10  Sergio Rodriguez
24  2014-15 *                Real Madrid  ...  21.48  Sergio Rodriguez
25  2014-15 *                Real Madrid  ...  20.10  Sergio Rodriguez
26  2015-16 *                  All Teams  ...  17.59  Sergio Rodriguez
27  2015-16 *                Real Madrid  ...  17.87  Sergio Rodriguez
28  2015-16 *                Real Madrid  ...  20.14  Sergio Rodriguez
29  2015-16 *                Real Madrid  ...   9.32  Sergio Rodriguez
30  2015-16 *                Real Madrid  ...  16.95  Sergio Rodriguez
31  2017-18 *                  All Teams  ...  19.19  Sergio Rodriguez
32  2017-18 *                CSKA Moscow  ...  19.36  Sergio Rodriguez
33  2017-18 *                CSKA Moscow  ...  17.89  Sergio Rodriguez
34  2018-19 *                  All Teams  ...  17.83  Sergio Rodriguez
35  2018-19 *                CSKA Moscow  ...  14.92  Sergio Rodriguez
36  2018-19 *                CSKA Moscow  ...  21.10  Sergio Rodriguez
37  2018-19 *                CSKA Moscow  ... -11.12  Sergio Rodriguez
38  2019-20 *                  All Teams  ...  18.23  Sergio Rodriguez
39  2019-20 *  EA7 Emporio Armani Milano  ...  18.26  Sergio Rodriguez
40  2019-20 *  EA7 Emporio Armani Milano  ...  18.06  Sergio Rodriguez

[41 rows x 41 columns]
https://basketball.realgm.com/player/Nikola-Jokic/Summary/49571
      Season                      Team       League  ...   DRtg    PER        Player
0  2012-13 *                 All Teams  All Leagues  ...   98.5  14.15  Nikola Jokic
1  2012-13 *  KK Mega Leks Junior Team     Belgrade  ...   95.2  21.13  Nikola Jokic
2  2012-13 *             KK Mega Bemax          KLS  ...  104.9   3.48  Nikola Jokic
3  2013-14 *                 All Teams  All Leagues  ...  108.5  19.42  Nikola Jokic
4  2013-14 *             KK Mega Bemax     Liga ABA  ...  108.4  21.18  Nikola Jokic
5  2013-14 *             KK Mega Bemax          KLS  ...  109.0  21.79  Nikola Jokic
6  2014-15 *                 All Teams  All Leagues  ...  100.6  24.61  Nikola Jokic
7  2014-15 *             KK Mega Bemax     Liga ABA  ...  100.5  24.02  Nikola Jokic
8  2014-15 *             KK Mega Bemax          KLS  ...  100.8  31.50  Nikola Jokic

[9 rows x 41 columns]
https://basketball.realgm.com/player/Brandon-Jennings/Summary/1609
      Season                    Team  ...    PER            Player
0  2008-09 *               All Teams  ...  13.05  Brandon Jennings
1  2008-09 *             Virtus Roma  ...  12.94  Brandon Jennings
2  2008-09 *             Virtus Roma  ...  12.43  Brandon Jennings
3    2017-18          Shanxi Zhongyu  ...  21.51  Brandon Jennings
4  2018-19 *               All Teams  ...  12.52  Brandon Jennings
5  2018-19 *  Zenit Saint Petersburg  ...  14.60  Brandon Jennings
6  2018-19 *  Zenit Saint Petersburg  ...  10.16  Brandon Jennings

[7 rows x 41 columns]
https://basketball.realgm.com/player/Thon-Maker/Summary/42192
No international table for Thon Maker.

Use html to go into url and scrape tables

Answers (1)

Related Questions