CODEWITHSUNDEEP
pythonseleniumwebdriver
Enrique Ramirez Moreno
Enrique Ramirez Moreno

Reputation: 37

Not clickable point (Selenium)

I'm trying to click the "Show more" button, but I can't.

Any help? Thank you very much.

from selenium import webdriver
import time

driver = webdriver.Chrome()
driver.get('https://www.scorespro.com/basketball/china/cba/results/')
time.sleep(2)
showmore = driver.find_element_by_link_text("Show more")
showmore.click()
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")``

Upvotes: 0

Views: 76

Answers (3)

chitown88
chitown88

Reputation: 28565

If you are after the tables, you don't need to use Selenium. you can pull the data straight away with requests, and parse with pandas.

To find that, when you go to the page, you want to right-click and 'Inspect' (or Shft-Ctrl-I). This will open a side panel. When it opens, you want to go to Network and XHR. And you want to sort of browse those requests (and you can click on Preview to see what it return. You may need to 1) reload the page; and 2) click around the table. For example, once I clicked "show more" at the bottom of the table, it popped up.

Once you find it, click on Headers and you'l see the url and payload, etc. I highlighted it in the pic for you:

enter image description here

import requests
import pandas as pd


url = 'https://www.scorespro.com/basketball/ajaxdata_more.php'
headers = {'User-Agent': 'Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.149 Mobile Safari/537.36'}


dfList = []
for season in ['2018-2019','2019-2020']:
    continueLoop = True
    page = 1
    while continueLoop == True:
        print ('%s Collecting Page: %s' %(season,page))
        payload = {
        'country': 'china',
        'comp': 'cba',
        'season': season,
        'status': 'results',
        'league': '',
        'page': '%s' %page}
    
    
        response = requests.get(url, headers=headers, params=payload)
        
        try:
            dfs = pd.read_html(response.text)
        except ValueError:
            print ('No more tables found.')
            continueLoop = False
            
        page+=1
        dfList.extend(dfs)

dfList_single = []
cols = ['Date','Final Time', 'Team', 'Final Score','Q1','Q2','Q3','Q4','OT','Half Time Score','Combined Total Score']
for each in dfList:
    each = each.loc[:,[0, 1, 2, 5, 6, 7, 8, 9, 10, 11, 12]]
    each.columns = cols
    teamA = each.iloc[0,:]
    teamB = each.iloc[1,2:]
    temp_df = pd.concat([teamA, teamB], axis=0).to_frame().T
    dfList_single.append(temp_df)


df = pd.concat(dfList_single)
df = df.reset_index(drop=True)

Output:

print(df.head(10).to_string())
             Date Final Time      Team  Final Score  Q1  Q2  Q3  Q4  Half Time Score  Combined Total Score
0  15.08.20 15:00         FT  Guandong          123  26  28  41  28               54                   238
1  15.08.20 15:00         FT  Liaoning          115  29  18  39  29               47                   238
2  13.08.20 15:00         FT  Liaoning          115  34  24  23  34               58                   228
3  13.08.20 15:00         FT  Guandong          113  38  28  27  20               66                   228
4  11.08.20 15:00         FT  Guandong          110  25  30  25  30               55                   198
5  11.08.20 15:00         FT  Liaoning           88  24  26  23  15               50                   198
6  08.08.20 15:00         FT  Guandong           88  16  21  26  25               37                   173
7  08.08.20 15:00         FT   Beijing           85  13  24  20  28               37                   173
8  07.08.20 15:00         FT  Liaoning          119  22  40  29  28               62                   232
9  07.08.20 15:00         FT  Xinjiang          113  33  22  34  24               55                   232

Upvotes: 1

rahul rai
rahul rai

Reputation: 2326

Two issue with your code: First you are trying to scroll after clicking, where as it should be before. Second You are using height of screen which may work in one device bit not in other if size varies. Better way is to scroll to element itself and the click. See below code. It worked fine:

import time
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC


driver = webdriver.Chrome('..\drivers\chromedriver')
driver.get("https://www.scorespro.com/basketball/china/cba/results/")
driver.maximize_window()
# I have accept cookies on page , so below step
WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.XPATH, "//a[text()='Agree']"))).click()
showMore = driver.find_element_by_link_text("Show more")
driver.execute_script("arguments[0].scrollIntoView();", showMore)
time.sleep(2)
showMore.click()

Upvotes: 0

Just for fun
Just for fun

Reputation: 4225

Dividing the height with some number can decrease the scroll height and will stop where Show More visible

from selenium import webdriver
import time

driver = webdriver.Chrome()
driver.get('https://www.scorespro.com/basketball/china/cba/results/')
driver.execute_script("window.scrollTo(0, document.body.scrollHeight/1.35);")
time.sleep(2)
driver.find_element_by_class_name("show_more").click()
time.sleep(2)
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")

Upvotes: 1

Related Questions