looping through a dropdown menu using Selenium and Python

Question

I'm trying to loop through a dropdown menu on at this url: https://www.accuform.com/safety-sign/danger-danger-authorized-personnel-only-MADM006

So, for example, the first dropdown menu - under options - lists out different materials and I want to select each one in turn and then gather some other information from the webpage before moving on to the next material. Here is my current code:

driver = webdriver.Firefox()
driver.get('https://www.accuform.com/safety-sign/danger-danger-authorized-personnel-only-MADM006')

time.sleep(3)

driver.find_element_by_id('x-mark-icon').click()

select = Select(driver.find_element_by_name('Wiqj7mb4rsAq9LB'))
options = select.options
optionsList = []

driver.find_elements_by_class_name('select-wrapper')[0].click()

element = driver.find_element_by_xpath("//select[@name='Wiqj7mb4rsAq9LB']")
actions = ActionChains(driver)
actions.move_to_element(element).perform()

# driver.execute_script("arguments[0].scrollIntoView();", element)


for option in options: #iterate over the options, place attribute value in list
    optionsList.append(option.get_attribute("value"))

for optionValue in optionsList:
    print("starting loop on option %s" % optionValue)
    # select = WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, "//select[@name='Wiqj7mb4rsAq9LB']")))
    # select = Select(select)
    select.select_by_value(optionValue)

I started with just the loop, but got this error:

ElementNotInteractableException: Message: Element <option> could not be scrolled into view

I then added the webdriverwait and get a TimeoutException error.

I then realized I should probably click on the wrapper in which the dropdown is held, so I added the click, which does pup up the menu, but I still got the TimeoutException.

So I thought, maybe I should move to the element, which I tried with the action chain lines and I got this error

WebDriverException: Message: TypeError: rect is undefined

I tried to avoid that error by using this code instead:

    # driver.execute_script("arguments[0].scrollIntoView();", element)

Which just resulted in the timeoutexception again.

I pretty new to Python and Selenium and have basically just been modifying code from SO answers to similar questions, but nothing has worked.

I'm using python 3.6 and the current versions of Selenium and firefox webdriver.

If anything is unclear or if you need more info just let me know.

Thanks so much!

EDIT: Based on the answer and comments by Kajal Kunda, I've updated my code to the following:

`material_dropdown = driver.find_element_by_xpath("//input[@class='select- 
dropdown']")

driver.execute_script("arguments[0].click();", material_dropdown)

materials=driver.find_elements_by_css_selector("div.select-wrapper 
ul.dropdown-content li")



for material in materials:

    # material_dropdown = 
    driver.find_element_by_xpath("//input[@class='select-dropdown']")

    # driver.execute_script("arguments[0].click();", material_dropdown)

    # materials=driver.find_elements_by_css_selector("div.select-wrapper ul.dropdown-content li")

    material_ele=material.find_element_by_tag_name('span')

if material_ele.text!='':

    material_ele.click()

    time.sleep(5)

    price = driver.find_element_by_class_name("dataPriceDisplay")

    print(price.text)`

The result is that it successfully prints the price for the first type of material, but then it returns: StaleElementReferenceException: Message: The element reference of <li class=""> is stale;...

I've tried variations of having the hashed out lines in and outside of the loop, but always get a version of the StaleElementReferenceException error.

Any suggestions?

Thanks!

Answer 1

Try the below code.It should work.

    driver = webdriver.Firefox()
    driver.get('https://www.accuform.com/safety-sign/danger-danger-authorized-personnel-only-MADM006')

    time.sleep(3)
    driver.find_element_by_id('x-mark-icon').click()

    material_dropdown = driver.find_element_by_xpath("//input[@class='select-dropdown']")
    driver.execute_script("arguments[0].click();", material_dropdown)

    #Code for material dropdown
    materials=driver.find_elements_by_css_selector("div.select-wrapper ul.dropdown-content li")


    material_optionsList = []
    for material in materials:
        material_ele=material.find_element_by_tag_name('span')
        if material_ele.text!='':
          material_optionsList.append(material_ele.text)

    print(material_optionsList)

    driver.execute_script("arguments[0].click();", material_dropdown)


    size_dropdown = driver.find_element_by_xpath("(//input[@class='select-dropdown'])[2]")
    driver.execute_script("arguments[0].click();", size_dropdown)

    #Code for size dropdown
    Sizes=driver.find_elements_by_css_selector("div.select-wrapper ul.dropdown-content li")
    size_optionsList = []
    for size in Sizes:
        size_ele=size.find_element_by_tag_name('span')
        if size_ele.text!='':
            size_optionsList.append(size_ele.text)



driver.execute_script("arguments[0].click();", size_dropdown)

Output :

[u'Adhesive Vinyl', u'Plastic', u'Adhesive Dura-Vinyl', u'Aluminum', u'Dura-Plastic\u2122', u'Aluma-Lite\u2122', u'Dura-Fiberglass\u2122', u'Accu-Shield\u2122']

Hope you will do the remaining.Let me know if it works for you.

EDIT Code for loop through and get the price value of materials.

for material in range(len(materials)):
    material_ele=materials[material]

    if material_ele.text!='':
       #material_optionsList.append(material_ele.text)
       #material_ele.click()
       driver.execute_script("arguments[0].click();", material_ele)
       time.sleep(2)
       price = driver.find_element_by_id("priceDisplay")
       print( price.text)
       time.sleep(2)
       material_dropdown = driver.find_element_by_xpath("//input[@class='select-dropdown']")
       driver.execute_script("arguments[0].click();", material_dropdown)
       materials = driver.find_elements_by_css_selector("div.select-wrapper ul.dropdown-content li")
       material+=2

Output :

$8.31
$9.06
$13.22
$15.91
$15.91

Answer 2

You could do the whole thing with requests. Grab the drop down list from the options listed in drop down then concatenate the value attributes into requests url that retrieves json containing all the info on the page. Same principle applies for adding in other dropdown values. The ids for each drop down selection are the value attributes of the options in the drop down and appear in the url I show separated by // for each drop down selection.

import requests
from bs4 import BeautifulSoup as bs

url = 'https://www.accuform.com/product/getSku/danger-danger-authorized-personnel-only-MADM006/1/false/null//{}//WHFIw3xXmQx8zlz//6wr93DdrFo5JV//WdnO0RpwKpc4fGF'
startURL = 'https://www.accuform.com/safety-sign/danger-danger-authorized-personnel-only-MADM006'

res = requests.get(startURL)
soup = bs(res.content, 'lxml')
materials = [item['value'] for item in soup.select('#Wiqj7mb4rsAq9LB option')]
sizes = [item['value'] for item in soup.select('#WvXESrTyQjM3Ciw option')]
languages = [item['value'] for item in soup.select('#WUYWGMePtpmpmhy option')]
units = [item['value'] for item in soup.select('#W91eqaJ0WPXwe9b option')]

for material in materials:
    data = requests.get(url.format(material)).json()
    soup = bs(data['dataMaterialBullets'], 'lxml')
    lines = [item.text for item in soup.select('li')]
    print(lines)
    print(data['dataPriceDisplay'])     
    # etc......

---

Sample of JSON: