CODEWITHSUNDEEP
pythonpython-3.xseleniumselenium-webdriverselenium-chromedriver
Kelvin Low
Kelvin Low

Reputation: 746

How do I deal with "Message: stale element reference: element is not attached to the page document" in Python Selenium

I'm writing a script to scrape product names from a website, filtered by brands. Some search results may contain more than one page, and this is where the problem comes in. I'm able to scrape the first page but when the script clicks on the next page the error message selenium.common.exceptions.StaleElementReferenceException: Message: stale element reference: element is not attached to the page document shows. Below is my code:

def scrape():
   resultList = []
   currentPage = 1

   while currentPage <= 2:
      titleResults = WebDriverWait(driver, 10).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, 'h4.mt-0')))
      resultList.append(titleResults)
      checkNextPage = WebDriverWait(driver, 30).until(EC.visibility_of_all_elements_located((By.XPATH, "//div/nav/ul/li/a[@aria-label='Next']")))
      for cnp in checkNextPage:
         nextPageNumber = int(cnp.get_attribute("data-page"))
      currentPage += 1
      driver.find_element_by_xpath("//div/nav/ul/li/a[@aria-label='Next']").click()
   for result in resultList[0]:
      print("Result: {}".format(result.text))

I think the error got triggered when .click() was called. I've done a lot of searching on the internet before resorting to posting this question here because either I don't understand the solutions from other articles/posts or they don't apply to my case.

Upvotes: 0

Views: 116

Answers (1)

Yun
Yun

Reputation: 1042

Stale Element means an old element or no longer available element.

I think the error is caused by last line.

You should extract elements text before the elements become unavailable.

def scrape():
   resultList = []
   currentPage = 1

   while currentPage <= 2:
      titleResults = WebDriverWait(driver, 
10).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, 'h4.mt-0')))

      // Extract elements text 
      results_text = [titleResults[i].text for i in range(0, len(titleResults))]
      resultList.extend(results_text)

      checkNextPage = WebDriverWait(driver, 30).until(EC.visibility_of_all_elements_located((By.XPATH, "//div/nav/ul/li/a[@aria-label='Next']")))
      for cnp in checkNextPage:
         nextPageNumber = int(cnp.get_attribute("data-page"))
      currentPage += 1
      driver.find_element_by_xpath("//div/nav/ul/li/a[@aria-label='Next']").click()


  print("Result: {}".format(resultList))

Upvotes: 1

Related Questions