Segmented scroll down page to check for existence of element - Python selenium, driver.execute_script

Question

I have a headache of a problem: I am trying to click the load more button of the skills section for Linkedin pages. But due to screen resolution, common solutions such as action chains that should allow me to scroll to the element, do not work.

I've posted about it here (where I was advised to post another question) and here (where the problem was proven to be screen resolution).

In both posts, the only solution that worked for me was a driver.execute_script("window.scrollTo(0, 1800)") type command, with specific coordinates for the specific Linkedin page I referenced as an example. The issue is that I need to loop this command for multiple profiles, so a specific coordinate on one page does not work on another profile, as the skills section is no longer located in the same place.

I've come up with a solution like this, but it seems like the worst possible solution:

try:
    ldmore=driver.find_element_by_xpath('//span[text()="Show more"]')
    ldmore.click()
except NoSuchElementException:
    driver.execute_script("scroll(0, 1600)")
    try:
        ldmore = driver.find_element_by_xpath('//span[text()="Show more"]')
        ldmore.click()
    except NoSuchElementException:
        driver.execute_script("scroll(0, 1800)")
        try:
            ldmore = driver.find_element_by_xpath('//span[text()="Show more"]')
            ldmore.click()
        except NoSuchElementException:
            driver.execute_script("scroll(0, 2000)")
            try:
                ldmore = driver.find_element_by_xpath('//span[text()="Show more"]')
                ldmore.click()
            except NoSuchElementException:
                driver.execute_script("scroll(0, 2200)")

There has to be a better way to do this rather than infinitely looping for every possible coordinate location. However, I'm at a loss as to what this solution could be.

EDIT: Many solutions use the line ldmore = driver.find_element_by_xpath('//span[text()="Show more"]') and then some syntax that scrolls to ldmore. The problem is that I receive Message: no such element: Unable to locate element in specifying the xpath, before the page is scrolled to the location. So, the code fails even before the following syntax can scroll to ldmore.

My relevant code for your reference:

ChromeOptions = webdriver.ChromeOptions()
driver = webdriver.Chrome('C:\\Users\\Root\\Downloads\\chromedriver.exe')

driver.get('https://www.linkedin.com/login?fromSignIn=true&trk=guest_homepage-basic_nav-header-signin')
driver.maximize_window()

WebDriverWait(driver, 10).until(
    EC.element_to_be_clickable((By.NAME, "session_key"))).send_keys("EMAIL")
WebDriverWait(driver, 10).until(
    EC.element_to_be_clickable((By.NAME, "session_password"))).send_keys("PASSWORD")
WebDriverWait(driver, 10).until(
    EC.element_to_be_clickable((By.XPATH, "//button[@class='btn__primary--large from__button--floating']"))).click()

driver.get("https://www.linkedin.com/in/kate-yun-yi-wang-054977127/?originalSubdomain=hk")
driver.maximize_window()

ldmore=driver.find_element_by_xpath('//span[text()="Show more"]')

EDIT: Attempted solution with break for the end of the page:

        looking_for_element = True
        ldmore = ''

        while looking_for_element:
            elements = driver.find_elements_by_xpath('//span[text()="Show more"]')

            if len(elements) > 0:
                ldmore = elements[0]
                sleep(5)
                ldmore.click()
                looking_for_element = False
            else:
                body = driver.find_element_by_css_selector('body')
                last_height = driver.execute_script("return document.body.scrollHeight")
                print(last_height)

                while True:

                    driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
                    time.sleep(2)
                    new_height = driver.execute_script("return document.body.scrollHeight")

                    driver.execute_script(str(last_height))

                    if new_height == last_height:
                        break

                    body.send_keys(Keys.PAGE_DOWN)

Answer 1

Instead of scrolling to a specific coordinate, you should be able to scroll to the element itself

driver.execute_script("arguments[0].scrollIntoView();", ldmore)

and then perform your click

ldmore.click()

or you can use javascript to click without scrolling

driver.execute_script('arguments[0].click();', ldmore)

In some cases where you are unable to find the element until you scroll far enough, I have found using a while loop and doing a page down until I find it works well.

looking_for_element = True
ldmore = ''

while looking_for_element:
    elements = driver.find_elements_by_xpath('//span[text()="Show more"]')

    if len(elements) > 0:
        ldmore = elements[0]
        looking_for_element = False
    else:
        body = driver.find_element_by_css_selector('body')
        body.send_keys(Keys.PAGE_DOWN)

you will need to add this, as well

from selenium.webdriver.common.keys import Keys

Ultimately, you will want to add some code to this loop to check for when the bottom of the page has been reached and the element has not been found that there is a way to break out of the loop.

Here is a way to exit the loop if the element is not found:

while looking_for_element:
    elements = driver.find_elements_by_xpath('//span[text()="Show more"]')

    if len(elements) > 0:
        ldmore = elements[0]
        looking_for_element = False
    else:
        global_copyright = driver.find_elements_by_css_selector('#globalfooter-copyright')

        if len(global_copyright) > 0:
            looking_for_element = False
        else:
            body = driver.find_element_by_css_selector('body')
            body.send_keys(Keys.PAGE_DOWN)

This is checking for the footer global copyright and once that element is found the loop knows that it has reached the bottom and will not continue with the page downs. I could have used a simple break instead of changing the 'looking_for_element' to False, but I think it's more readable this way.