CODEWITHSUNDEEP
pythonseleniumexceptionweb-crawlerwait
신종원
신종원

Reputation: 195

How can I wait until a specific drop-down menu selected using Selenium Python?

I made a crawler for this page (http://www.bobaedream.co.kr/cyber/CyberCar.php?gubun=I) to collect the stock list of specific manufacturers. The process is to start from selecting the drop-down menu in the first row of upper part of search menu.

enter image description here

Each right drop-down menu is child menu of its left drop-down menu. What I would like to do is to select each first item in each drop-down menu and click the "search" button for the first run. After crawling of its stock list, then I set the second item of the last drop-down menu and click the "search" button.

But the problem is occurred here. I saved each items of each drop-down menu as tuple. When I try to call the second item of the last drop-down menu for the second round of crawling, "StaleElementReferenceException" or "NoSuchElementException" is occurred with the message of "Element is no longer attached to the DOM". Thus, I would like to make the element wait until the entire round of each drop-down iteration is completed.

Below is my code, but still have the error message. My error usually occurs at the second while loop. At this moment, I guess some type of "wait.until(EC.~)" code in the second "try" function can work this out, but I have no specific idea for this. Please help or give me any advice. 

def option2_menu_loaded(inDriver):
    path = '//select[@id="level2_no"]'
    return inDriver.find_element_by_xpath(path)

self.wait.until(option2_menu_loaded)

while True:
    try:
        select_option2_values = [
            ('%s' % o.get_attribute('text'), '%s' % o.get_attribute('value'))
            for o
            in self.getNewSelect("#level2_no").options
            if o.get_attribute('text') != '세부등급']
    except (StaleElementReferenceException, NoSuchElementException):
        print("Exception Found")
        continue
    break

for option2 in select_option2_values:
    self.csv.setCarTitle(ma, mo, de, option1[0], option2[0])

    print(option2[0], option2[1])
    self.driver.implicitly_wait(0.5)

    while True:
        try:
            self.getNewSelect("#level2_no").select_by_value(option2[1])
        except (StaleElementReferenceException, NoSuchElementException):
            self.getNewSelect("#level2_no").options
            print("Exception Found")
            continue
        break

Upvotes: 0

Views: 858

Answers (1)

Xwris Stoixeia
Xwris Stoixeia

Reputation: 1861

If you google the StaleElementException you will see solutions that try to find again the element within a loop. So that is one idea, in your exception above try 3 times with 1 sec delay before each try to find_Element again, see if this helps.

Another idea is to refresh the page (certainly not ideal but it might work) between every crawl. You can do this in Python using:

driver.refresh()

Finally, you can also avoid looping (this might be causing the StaleElementException) through all the different elements when crawling as Selenium has a solution for that. You can save eveything in tuple/array without looping through each record by using find_ElementS instead of find_ElemenT. Try this see if it improves your overall performance:

a=[];
a = driver.find_elements_by_xpath(path)

Best of luck!

Upvotes: 1

Related Questions