CODEWITHSUNDEEP
pythonhtmlseleniumselenium-webdriverweb-scraping
user11669928
user11669928

Reputation:

How to wait till page loads and then press the load more button in selenium?

Using selenium to load and page and need to click load more button, but after it clicks load more than 100 times

i m getting this error of element click intercepted. because after 100 times the page takes time to load. and code doesnt know where to click.

Tried increasing sleep time to 20 seconds also but at some points if page takes more than 20 seconds the code returns error

error : 


ElementNotInteractableException: Message: element not interactable (Session info: chrome=75.0.3770.100)

code : 

from selenium import webdriver
import time
import pandas as pd
driver = webdriver.Chrome('/Users/1/chromedriver.exe')
driver.get('https://simpletire.com/catalog?select=1&brand=61&query=catalog')

click_more=True
while click_more:
    time.sleep(2)

    driver.find_element_by_css_selector(".btn.btn-primary.btn-lg").click();

Upvotes: 0

Views: 2307

Answers (4)

SIM
SIM

Reputation: 22440

Try the script below to keep clicking on that button until there is none left to click.

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.support import expected_conditions as EC

with webdriver.Chrome() as driver:
    wait = WebDriverWait(driver, 10)
    driver.get('https://simpletire.com/catalog?select=1&brand=61&query=catalog')

    while True:
        try:
            item = wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, "#load_button > button")))
        except TimeoutException: 
            item = ""

        if not item: break

        driver.execute_script("arguments[0].scrollIntoView();", item)
        item.click()

Upvotes: 0

Marcel Wilson
Marcel Wilson

Reputation: 4572

It would appear that you might be running into the google ad at the bottom of the page. 

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException

driver = webdriver.Chrome()
driver.get('https://simpletire.com/catalog?select=1&brand=61&query=catalog')

click_more=True
wait = WebDriverWait(driver, 30)
while click_more:
    try:
        elem = wait.until(EC.visibility_of_element_located((By.XPATH, '//button[@class="btn btn-primary btn-lg"]')), 
                          "Cannot find 'Load More' button")
        elem.click()
    except TimeoutException:
        click_more = False

Using xpath and visibility_of_element_located I get the following exception:

selenium.common.exceptions.ElementClickInterceptedException: Message: element click intercepted: Element <button class="btn btn-primary btn-lg">...</button> is not clickable at point (591, 797). Other element would receive the click: <iframe id="google_ads_iframe_/21692090825/ST_FlexFooter_0" title="3rd party ad content" name="google_ads_iframe_/21692090825/ST_FlexFooter_0" width="970" height="90" scrolling="no" marginwidth="0" marginheight="0" frameborder="0" srcdoc="" style="border: 0px; vertical-align: bottom;" data-google-container-id="1" data-load-complete="true"></iframe>

You either need to close the google add or scroll the page down a little before attempting to click the button. Once that is done, the while loop should work.

Upvotes: 0

Mike
Mike

Reputation: 26

In these situations I typically stop the page load, perhaps scroll a couple of times and use screen coordinates with a macro tool like AppRobotic to click the button. Something like this should work for you:

import win32com.client
from win32com.client import Dispatch
x = win32com.client.Dispatch("AppRobotic.API")
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

driver = webdriver.Chrome('/Users/1/chromedriver.exe')
driver.get('https://simpletire.com/catalog?select=1&brand=61&query=catalog')
# wait 20 seconds per question
x.Wait(20000)
# scroll down a couple of times in case page javascript is waiting for user interaction
x.Type("{DOWN}")
x.Wait(2000)
x.Type("{DOWN}")
x.Wait(2000)
# forcefully stop pageload at this point
driver.execute_script("window.stop();")
# if clicking with Selenium still does not work here, use screen coordinates
x.MoveCursor(xCoord, yCoord)
x.MouseLeftClick
x.Wait(2000)

Upvotes: 0

Dmitri T
Dmitri T

Reputation: 167992

Consider introducing Explicit Wait to ensure that the button is there prior to attempting clicking it

Example code:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

driver = webdriver.Chrome('/Users/1/chromedriver.exe')
driver.get('https://simpletire.com/catalog?select=1&brand=61&query=catalog')

click_more=True
while click_more:
    WebDriverWait(driver, 30).until(EC.presence_of_element_located((By.CSS_SELECTOR, ".btn.btn-primary.btn-lg"))).click()

More information:

Upvotes: 2

Related Questions