CODEWITHSUNDEEP
pythonseleniumwebdriver
user10830454
user10830454

Reputation:

Python - Click to scroll to bottom using Selenium

I'm using Selenium in Python to extract a list of startups from AngelList. In order to collect all startups, I have to click the button 'More' at the end of the page until I reach the end of the list.

The problem is I cannot figure out how to keep clicking until it reaches the end of the page. 

driver = webdriver.Chrome('C:\\Users\\Documents\\chromedriver.exe')
driver.get("https://angel.co/companies?company_types[]=Startup")
driver.find_element_by_class_name("""more""").click()

This results in one click of "More". Each click loads 20 more startups.

I have tried this in order to keep clicking:

i = 0
while i < 20:
    driver.find_element_by_class_name("""more""").click()
    i += 1

and it results in this error:

selenium.common.exceptions.StaleElementReferenceException: Message: stale 
element reference: element is not attached to the page document

Any help is much appreciated.

Upvotes: 2

Views: 980

Answers (1)

ewwink
ewwink

Reputation: 19164

when it reaches the end of the page, the element <div class="more">More</div> will be removed from the DOM.

To click and load more content, wait and check if the button or div.more has text More, here the example using WebDriverWait and filtered results URL

from selenium.webdriver.support.ui import WebDriverWait

driver.get('https://angel.co/companies?company_types[]=Startup&markets[]=Education&raised[min]=2830196&raised[max]=100000000&stage[]=Series+B&stage[]=Series+A')

while True:
    try:
        moreButton = WebDriverWait(driver, 10).until(
            lambda d: d.find_element_by_xpath('//div[@class="more" and text() = "More"]')
        )
        moreButton.click()
    except:
        print("scroll finished")
        break

Upvotes: 2

Related Questions