CODEWITHSUNDEEP
pythonseleniumweb-scraping
Aditya
Aditya

Reputation: 571

Selenium error: Element not found in the cache - perhaps the page has changed since it was looked up

I am extracting the first 'name' field on each page of the url: "http://www.srlworld.com/content/65/find-a-lab.html"

The for loop runs once and throws an error:

File "srl.py", line 40, in <module>
    print state.text
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webelement.py", line 66, in text
    return self._execute(Command.GET_ELEMENT_TEXT)['value']
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webelement.py", line 404, in _execute
    return self._parent.execute(command, params)
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 195, in execute
    self.error_handler.check_response(response)
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/errorhandler.py", line 170, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.StaleElementReferenceException: Message: Element not found in the cache - perhaps the page has changed since it was looked up
Stacktrace:
    at fxdriver.cache.getElementAt (resource://fxdriver/modules/web-element-cache.js:8981)
    at Utils.getElementAt (file:///tmp/tmpPEHToH/extensions/[email protected]/components/command-processor.js:8574)
    at WebElement.getElementText (file:///tmp/tmpPEHToH/extensions/[email protected]/components/command-processor.js:11722)
    at DelayedCommand.prototype.executeInternal_/h (file:///tmp/tmpPEHToH/extensions/[email protected]/components/command-processor.js:12282)
    at fxdriver.Timer.prototype.setTimeout/<.notify (file:///tmp/tmpPEHToH/extensions/[email protected]/components/command-processor.js:603)

The code is:

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import Select

driver = webdriver.Firefox()
driver.get("http://www.srlworld.com/content/65/find-a-lab.html")
#assert "http" in driver.title

elem = driver.find_element_by_id("country")
#driver.implicitly_wait(5)
all_countries = elem.find_elements_by_tag_name("option")
country = all_countries[1]
print "country value is %s" % country.get_attribute("value")
country.click()
driver.implicitly_wait(2)

state_elem = driver.find_element_by_id("state")
all_states = state_elem.find_elements_by_tag_name("option")
del all_states[0]


for state in all_states:
    print "start ",
    print state.text

    print "state value is %s" % state.get_attribute("value")
    state.click()
    driver.implicitly_wait(2)

    driver.find_element_by_name("go").click()

    name = driver.find_element_by_xpath("//div[span='Name'][1]/span/following-sibling::span[2]")
    print name.text
    print "end ",
    print state.text

On running this script, the for loop which runs only once doesn't print the last 'state.text' even though I am not making any changes.

Upvotes: 3

Views: 3139

Answers (1)

vadimhmyrov
vadimhmyrov

Reputation: 189

Considering the text of the exception, the following happens: each time you press the button "Go" the page refreshes itself (loads new data, not by AJAX, but by actually refreshing - this is important), therefore Selenium detects the page state change and throws an exception as you try to access elements from its previous state. I suggest the following algorithm to solve your problem:

current_position = 1

while True:
    try:
        state_elem = driver.find_element_by_id("state")
        all_states = state_elem.find_elements_by_tag_name("option")
        state = all_states[current_position]
        print "start ",
        print state.text

        print "state value is %s" % state.get_attribute("value")
        state.click()
        driver.implicitly_wait(2)

        driver.find_element_by_name("go").click()

        name = driver.find_element_by_xpath("//div[span='Name'][1]/span/following-sibling::span[2]")
        print name.text
        print "end ",
        print state.text
        current_position += 1
    except:
        break

This way, you select each time the next option on a freshly generated page, and you shouldn't get the exception you had before.

Upvotes: 1

Related Questions