CODEWITHSUNDEEP
pythonweb-scrapingselenium-chromedriver
Véronique Lamontagne
Véronique Lamontagne

Reputation: 47

How to avoid StaleElementReferenceError when getting elements from different page?

I want to get all the results from a race. The website shows 50 rows/page. I navigate to the next page (same URL with suffix #page-x) using selenium, but I get a StaleElementReferenceException error whenever I try to find elements (cells of the table = td) on the next page.

I tried to close the driver between the steps to get just one list of elements at a time. I've also tried to load the pages separately with the URL+suffix, but it doesn't load correctly. I've tried building separate lists (at first I wanted one big list with all the results).

from selenium import webdriver
url = "https://tickets.justrun.ca/quidchrono.php?a=qcResult&raceid=8444"

#The block under works well and I get a list of cells as intended.
driver = webdriver.Chrome()
driver.maximize_window()
driver.get(url)
elements = driver.find_elements_by_tag_name("td")
course = []
for i in range(len(elements)):
    course.append(elements[i].text)

to_2 = driver.find_element_by_link_text("2")
to_2.click()
print(driver.current_url)

#I'm trying similar code for the next chunk, but it doesn't work.
elements2 = driver.find_elements_by_tag_name("td")
print(len(elements2))
print(elements2[5].text)
course2 = []
for i in range(len(elements2)):
    course2.append(elements2[i].text)
driver.close()

I would expect a new list (course2), with the results of the second page, but I get a stale element error. When I print the current URL, the result is as expected. When I print the len(elements2), it's also OK. Looks like the problem is when I try to get the text of an element.

Upvotes: 1

Views: 114

Answers (1)

bharatk
bharatk

Reputation: 4315

Solution-1:

Using BeautifulSoup and selenium, WebDriverWait is waiting for a certain condition to occur before proceeding further in the code. for more details about BeautifulSoup.

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from bs4 import BeautifulSoup

url = "https://tickets.justrun.ca/quidchrono.php?a=qcResult&raceid=8444"
driver = webdriver.Chrome()

driver.get(url)

data = []
while True:
    course = []
    WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.CLASS_NAME, "tableJustrun")))

    page_soup = BeautifulSoup(driver.page_source, 'lxml')
    # get table data 
    tbody = page_soup.find("tbody",{"id":"searchResultBoxParticipants"})
    rows = tbody.find_all("tr")

    for row in rows:
        rowData = []
        for td in row.find_all("td"):
            rowData.append(td.text)

        course.append(rowData)
    data.append(course)

    try:
        pagination = driver.find_element_by_class_name("simple-pagination")
        next_page = pagination.find_element_by_link_text("Suivant")
        # iterate next page
        next_page.click()
    except Exception as e:
        break

print(data)

Solution-2:

Using pandas library. 

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import pandas as pd

url = "https://tickets.justrun.ca/quidchrono.php?a=qcResult&raceid=8444"
driver = webdriver.Chrome()
driver.get(url)

data = []
while True:
    WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.CLASS_NAME, "tableJustrun")))
    tables = pd.read_html(driver.page_source)
    #append Participants table data
    data.append(tables[0])

    try:
        pagination = driver.find_element_by_class_name("simple-pagination")
        next_page = pagination.find_element_by_link_text("Suivant")
        # iterate next page
        next_page.click()
    except Exception as e:
        break

#Concat dataframe object
result = pd.concat(data)
print(result)

Upvotes: 1

Related Questions