CODEWITHSUNDEEP
pythonpython-3.xseleniumselenium-webdriverweb-scraping
SIM
SIM

Reputation: 22440

Script fails to keep clicking on load more button

I've written a script in Python in association with selenium to keep clicking on MORE button to load more items until there are no new items left to load from a webpage. However, my below script can click once on that MORE button available on the bottom of that page.

Link to that site

This is my try so far:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

link = "https://angel.co/companies?company_types[]=Startup&company_types[]=Private+Company&company_types[]=Mobile+App&locations[]=1688-United+States"

driver = webdriver.Chrome()
wait = WebDriverWait(driver, 10)
driver.get(link)
while True:
    for elems in wait.until(EC.presence_of_all_elements_located((By.CSS_SELECTOR,".results .name a.startup-link"))):
        print(elems.get_attribute("href"))

    try:
        loadmore = wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR,"[class='more']")))
        driver.execute_script("arguments[0].scrollIntoView();", loadmore)
        loadmore.click()
    except Exception:break

driver.quit()

How can I keep clicking on that MORE button until there are no such button left to click and parse the links as I've already tried using for loop.

Upvotes: 1

Views: 617

Answers (2)

SIM
SIM

Reputation: 22440

I've managed to solve the problem pursuing sir Andersson's logic within my exising script. This is what the modified script look like.

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

link = "https://angel.co/companies?company_types[]=Startup&company_types[]=Private+Company&company_types[]=Mobile+App&locations[]=1688-United+States"

driver = webdriver.Chrome()
wait = WebDriverWait(driver, 10)
driver.get(link)
while True:
    try:
        loadmore = wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR,"[class='more']")))
        driver.execute_script("arguments[0].click();", loadmore)
        wait.until(EC.staleness_of(loadmore))
    except Exception:break

for elems in wait.until(EC.presence_of_all_elements_located((By.CSS_SELECTOR,".results .name a.startup-link"))):
    print(elems.get_attribute("href"))

driver.quit()

Upvotes: 2

angrybambr
angrybambr

Reputation: 128

why not just?

        while (driver.FindElements(By.ClassName("more")).Count > 0)
        {
            driver.FindElement(By.ClassName("more")).Click();
            //Some delay to wait lazyload to complete 
        }

c# example. pretty sure that it can be done with python as well

Upvotes: 0

Related Questions