CODEWITHSUNDEEP
pythonselenium
Hugo Iriarte
Hugo Iriarte

Reputation: 65

Python Selenium Stale element reference

I know there are many questions about this but none seemed to help. Using keys to open a new tab or window so elements don't become stale does nothing. doesn't open a new tab. It does nothing. I am scraping a website that the main page has a list of links that I need to loop through click each and get data from each link. After the first iteration when it goes back to the main page elements are now stale obviously. This is my first time using Selenium so I apologize for this naive question but I've been stuck for days now and I'm desperate at this point for some help. Here is my original code: from selenium import webdriver

#Main url with a list on links of doctors name
url = ('https://weedmaps.com/doctors/in/massachusetts')


chromedriver = 'C:/Users/Henry/Desktop/chromedriver'
driver = webdriver.Chrome(chromedriver)

#Get the URL
driver.get(url)

#The element list
elements = driver.find_elements_by_class_name('details')

#Data I need
name = []
number = []
address = []
email = []
website = []

#Loop through elements
for i in elements:
    #clicks the element
    i.click()
    #New element once first element is clicked
    popup = driver.find_element_by_class_name('no-decoration')
    #Click the new element that opens a new window with the data I need
    popup.click()
    driver.implicitly_wait(10)
    #Data I need
    anchors = driver.find_elements_by_tag_name('a')
    address.append(anchors[29].text)
    number.append(anchors[30].text)
    email.append(anchors[31].text)
    website.append(anchors[35].text)
    #Go back to the list and go to the next element
    driver.back()

Upvotes: 0

Views: 475

Answers (1)

Andersson
Andersson

Reputation: 52685

You're making actions that updates the DOM, so elements that valid on first iteration is no more valid (becomes stale) on next iteration...

Instead of 

elements = driver.find_elements_by_class_name('details')
for i in elements:
    i.click()

try

elements_len = len(driver.find_elements_by_class_name('details'))
for i in range(elements_len):
    driver.find_elements_by_class_name('details')[i].click()

Upvotes: 0

Related Questions