CODEWITHSUNDEEP
pythonselenium-webdriverweb-scrapingnosuchelementexception
Mikel Pers
Mikel Pers

Reputation: 1

Why does my script trip the NoSuchElement exception when I know the element exists?

I wrote a script in which I defined a 'next page' function which scrolls to the bottom of the page, identifies the 'next' button and then clicks the button to navigate to the next page. See the following excerpt from my script:

def next_page():
    browser.execute_script("window.scrollTo(0,document.body.scrollHeight);")
    sleep(1)

    try:
        next_button = browser.find_element(By.CLASS_NAME, 'bbsPager_next')
        next_button.click()

    except NoSuchElementException:
        print("Next button not found. End of pagination or element not loaded yet.")
        return False

    return True

Later in my script, I call this function as part of a while True loop, but it keeps throwing the NoSuchElement Exception. The odd thing is I know the element exists, and in fact, I am able to successfully navigate to the next page when I manually call the next_page() function. So I know the function works. I just can't figure out why it is not working when integrated as part of the overall script.

My overall goal was to create a web scraper using Python Selenium that extracts relevant data fields from the current page of the target website, then automatically navigates to the next page and repeats the same process until it has extracted data from all of the pages available. The issue I am having, as discussed above, is that my 'while True' loop keeps breaking as a result of the NoSuchElement exception being thrown while executing the next_page() function. The following is an excerpt of the while True loop I am referring to:

while True:
    extract_page_data()
    navigation_successful = next_page()
    if not navigation_successful:
        print('No more pages left. Exiting...')
        break

I expected it to extract data from the page, navigate to the next page and repeat the process all over again until there are no more pages left. However, the output I receive is as follows:

"Next button not found. End of pagination or element not loaded yet. No more pages left. Exiting..."

Another important piece of information is that when I run the script, it successfully extracts information from page 1, and it even successfully navigates to page 2 before throwing the error. The weird part is that the error is thrown before data is extracted from the second page. This is unexpected, as I would have thought that if the script successfully navigates to the next page, the loop should repeat itself beginning with calling the data extraction function. It's almost as if the next page function is successfully performing its function while throwing an error at the same time. Is this possible? If so, how do I fix the code?

Upvotes: 0

Views: 51

Answers (1)

Saurabh Mudgal
Saurabh Mudgal

Reputation: 36

I would need to see the rest of your script to diagnose the issue with certainty, since you said running the function manually works.

However, one potential issue I can foresee is that the element may not necessarily load and become clickable yet, when you run it in a while True loop. Try modifying your function like this:

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

def next_page(browser):
    # Scroll to bottom of page in order for 'Next' button to be visible
    browser.execute_script("window.scrollTo(0, document.body.scrollHeight);")
    sleep(1)

    try:
        # Wait for the 'Next' button to be clickable
        next_button = WebDriverWait(browser, 10).until(
            EC.element_to_be_clickable((By.CLASS_NAME, 'bbsPager_next'))
        )
        next_button.click()
        return True

    except NoSuchElementException as e:
        print(f"Next button not found. End of pagination or element not loaded yet. Details: {e}")
        return False
    except Exception as e:
        print(f"Unknown exception occured: {e}")
        return False

Upvotes: 0

Related Questions