CODEWITHSUNDEEP
pythonpython-3.xseleniumselenium-webdriver
Abhay Salvi
Abhay Salvi

Reputation: 1119

StaleElementReferenceException Using Selenium in Python

Yes i know that this type of question has been answered many times before, but none of them helped me. Actually i didn't know much about it so need your help!

My problem: I am scraping through a website and it needs a CAPTCHA to search for every input. So i use FireFox as my browser as it asks for the captcha one time and doesn't change it. My code asks the user for CAPTCHA one time and then click on search button and tries to scrape the data, but when it clicks on the search button again (as it is in a loop) then it raises this error:

selenium.common.exceptions.StaleElementReferenceException: 
Message: The element reference of <input id="txt_ALPHA_NUMERIC" class="ui-inputfield ui-inputtext ui-widget ui-state-default ui-corner-all" name="txt_ALPHA_NUMERIC" type="text"> is stale; 
either the element is no longer attached to the DOM, it is not in the current frame context, or the document has been refreshed

My old code:

from selenium import webdriver                    # Import module 
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys   # For keyboard keys 
import time    
import pandas as pd

URL = 'https://vahan.nic.in/nrservices/faces/user/searchstatus.xhtml'      # Define URL 
browser = webdriver.Firefox(executable_path=r'C:\Users\intel\Downloads\Setups\geckodriver.exe')  

browser.get(URL)

vehicle_no = browser.find_element_by_xpath("""//*[@id="regn_no1_exact"]""")
vehicle_no.send_keys('RJ14CX3238')

captcha_input = input("enter your captcha ")
captcha = browser.find_element_by_xpath("""//*[@id="txt_ALPHA_NUMERIC"]""")
captcha.send_keys(captcha_input)
button_click = browser.find_element_by_xpath("/html/body/form/div[1]/div[3]/div/div[2]/div/div/div[2]/div[5]/div/button/span").click()    

i = 111
attempt = 1
max_attempts = 2
while True: 
    i = i + 1
    time.sleep(4)
    reg_no = browser.find_element_by_xpath("/html/body/form/div[1]/div[3]/div/div[2]/div/div/div[2]/div[6]/div/div/div/table/tbody/tr[2]/td[2]/span").text
    date = browser.find_element_by_xpath("/html/body/form/div[1]/div[3]/div/div[2]/div/div/div[2]/div[6]/div/div/div/table/tbody/tr[2]/td[4]").text 

    vehicle_no = browser.find_element_by_xpath("""//*[@id="regn_no1_exact"]""")
    vehicle_no.send_keys('RJ14CX3' + str(i))

    captcha.send_keys(captcha_input)

    button_click = browser.find_element_by_xpath("/html/body/form/div[1]/div[3]/div/div[2]/div/div/div[2]/div[5]/div/button/span").click()    
    browser.execute_script("return arguments[0].scrollIntoView(true);", button_click)

Updated new code now:

from selenium import webdriver                    # Import module 
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys   # For keyboard keys 
import time    
import pandas as pd
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

URL = 'https://vahan.nic.in/nrservices/faces/user/searchstatus.xhtml'      # Define URL 
browser = webdriver.Firefox(executable_path=r'C:\Users\intel\Downloads\Setups\geckodriver.exe')  

browser.get(URL)

vehicle_no = browser.find_element_by_xpath("""//*[@id="regn_no1_exact"]""")
vehicle_no.send_keys('RJ14CX3238')

captcha_input = input("enter your captcha ")
captcha = browser.find_element_by_xpath("""//*[@id="txt_ALPHA_NUMERIC"]""")
captcha.send_keys(captcha_input)
button_click = browser.find_element_by_xpath("/html/body/form/div[1]/div[3]/div/div[2]/div/div/div[2]/div[5]/div/button/span").click()    

i = 111
while True: 
    button_click = browser.find_element_by_xpath("/html/body/form/div[1]/div[3]/div/div[2]/div/div/div[2]/div[5]/div/button/span")
    WebDriverWait(browser, 10).until_not(EC.visibility_of_element_located((By.ID, "overley")))
    browser.execute_script("return arguments[0].scrollIntoView(true);", button_click)

    i = i + 1
    #reg_no = browser.find_element_by_xpath("/html/body/form/div[1]/div[3]/div/div[2]/div/div/div[2]/div[6]/div/div/div/table/tbody/tr[2]/td[2]/span").text
    #date = browser.find_element_by_xpath("/html/body/form/div[1]/div[3]/div/div[2]/div/div/div[2]/div[6]/div/div/div/table/tbody/tr[2]/td[4]").text 
    time.sleep(5)
    vehicle_no.send_keys('RJ14CX3' + str(i))
    WebDriverWait(browser, 10).until_not(EC.visibility_of_element_located((By.ID, "overley")))
    captcha.send_keys(captcha_input)

Also fix any other problems if is in my code. Any help would be appreciated!! Thanks in advance.

Upvotes: 0

Views: 155

Answers (1)

Asyranok
Asyranok

Reputation: 985

Simply re-find the button element in the loop, each time, rather than before the loop starts. Any time the DOM mutates, previous references are marked as stale, and will require a new instance. Interacting with Captcha's mutate the DOM, and mark the page as dirty (having changed/modified), which Selenium uses to flag "staleness".

Upvotes: 1

Related Questions