CODEWITHSUNDEEP
pythonseleniumstaleelementreferenceexception
Sukaldi
Sukaldi

Reputation: 13

StaleElementReferenceException while looping over list

I'm trying to make a webscraper for this website. The idea is that code iterates over all institutions by selecting the institution's name (3B-Wonen at first instance), closes the pop-up screen, clicks the download button, and does it all again for all items in the list.

However, after the first loop it throws the StaleElementReferenceException when selecting the second institution in the loop. From what I read about it this implies that the elements defined in the first loop are no longer accessible. I've read multiple posts but I've no idea to overcome this particular case.

Can anybody point me in the right directon? Btw, I'm using Pythons selenium and I'm quite a beginner in programming so I'm still learning. If you could point me in a general direction that would help me a lot! The code I have is te following:

#importing and setting up parameters for geckodriver/firefox
...

# webpage
driver.get("https://opendata-dashboard.cijfersoverwonen.nl/dashboard/opendata-dashboard/beleidswaarde")

WebDriverWait(driver, 30)

# Get rid of cookie notification
# driver.find_element_by_class_name("cc-compliance").click()

# Store position of download button
element_to_select = driver.find_element_by_id("utilsmenu")
action = ActionChains(driver)
WebDriverWait(driver, 30)

# Drop down menu
driver.find_element_by_id("baseGeo").click()

# Add institutions to array
corporaties=[]
corporaties = driver.find_elements_by_xpath("//button[@role='option']")

# Iteration
for i in corporaties:
    i.click()                                                     # select institution
    driver.find_element_by_class_name("close-button").click()     # close pop-up screen
    action.move_to_element(element_to_select).perform()           # select download button
    driver.find_element_by_id("utilsmenu").click()                # click download button
    driver.find_element_by_id("utils-export-spreadsheet").click() # pick export to excel
    driver.find_element_by_id("baseGeo").click()                  # select drop down menu for next iteration

Upvotes: 1

Views: 50

Answers (2)

pmadhu
pmadhu

Reputation: 3433

This code worked for me. But I am not doing driver.find_element_by_id("utils-export-spreadsheet").click()

from selenium import webdriver
import time
from selenium.webdriver.common.action_chains import ActionChains

driver = webdriver.Chrome(executable_path="path")
driver.maximize_window()
driver.implicitly_wait(10)

driver.get("https://opendata-dashboard.cijfersoverwonen.nl/dashboard/opendata-dashboard/beleidswaarde")

act = ActionChains(driver)
driver.find_element_by_xpath("//a[text()='Sluiten en niet meer tonen']").click() # Close pop-up
# Get the count of options
driver.find_element_by_id("baseGeoContent").click()
cor_len = len(driver.find_elements_by_xpath("//button[contains(@class,'sel-listitem')]"))
print(cor_len)
driver.find_element_by_class_name("close-button").click()

# No need to start from 0, since 1st option is already selected. Start from downloading and then move to next items.
for i in range(1,cor_len-288): # Tried only for 5 items
    act.move_to_element(driver.find_element_by_id("utilsmenu")).click().perform()
    #Code to click on downloading option
    print("Downloaded:{}".format(driver.find_element_by_id("baseGeoContent").get_attribute("innerText")))
    driver.find_element_by_id("baseGeoContent").click()
    time.sleep(3) # Takes time to load.
    coritems = driver.find_elements_by_xpath("//button[contains(@class,'sel-listitem')]")
    coritems[i].click()
    driver.find_element_by_class_name("close-button").click()
driver.quit()

Output:

295
Downloaded:3B-Wonen
Downloaded:Acantus
Downloaded:Accolade
Downloaded:Actium
Downloaded:Almelose Woningstichting Beter Wonen
Downloaded:Alwel

Upvotes: 1

cruisepandey
cruisepandey

Reputation: 29362

Problem Explanation :

See the problem here is, that you have defined a list corporaties = driver.find_elements_by_xpath("//button[@role='option']") and then iterating this list, and clicking on first element, which may cause some redirection to a new page, or in a new tab etc.

so when Selenium try to interact with the second webelement from the same list, it has to come back to original page, and the moment it comes back, all the elements become stale in nature.

Solution :

one of the basic solution in this cases are to define the list again, so that element won't be stale. Please see the illustration below :-

Code :

corporaties=[]
corporaties = driver.find_elements_by_xpath("//button[@role='option']")
# Iteration
j = 0
for i in range(len(corporaties)):
    elements = driver.find_elements_by_xpath("//button[@role='option']")
    elements[j].click()
    j = j + 1 # select institution
    driver.find_element_by_class_name("close-button").click()     # close pop-up screen
    action.move_to_element(element_to_select).perform()           # select download button
    driver.find_element_by_id("utilsmenu").click()                # click download button
    driver.find_element_by_id("utils-export-spreadsheet").click() # pick export to excel
    driver.find_element_by_id("baseGeo").click()  # select drop down menu for next iteration
    time.sleep(2)

Upvotes: 0

Related Questions