CODEWITHSUNDEEP
pythonseleniumweb-scraping
Mottey
Mottey

Reputation: 3

Python - Selenium - Cycling through drop-down menu - Option not visible

Context: I'm trying to retrieve information concerning 2015 municipal electoral data in Colombia from this webpage : https://elecciones.registraduria.gov.co:81/esc_elec_2015/99AL/DAL01004ZZZZZZZZZZZZ_L1.htm

As a starting point, I have already succesfully scraped the number of registered voters ("personas habilitadas") in the municipality of Abejorral using BeautifulSoup (15.152). The next step is to do the same for all municipalities ("municipios").

In order to do this, I'm trying to get around Selenium in Python 3.5 to cycle through the "Municipio" drop-down menu so I can select each municipio one by one and retrieve the corresponding number for "personas habilitadas". The issue I'm currently faced with is that I can't manage to tell Selenium to select another option/municipality in the drop-down menu, for example "Abriaqui". The corresponding HTML code is here (list of options purposedly shortened):

<select id="combo3" class="chosen-select" style="display: none;">
   <option value="../99AL/DAL01ZZZZZZZZZZZZZZZ_L1.htm">Todos</option>
   <option selected="selected" value="../99AL/DAL01004ZZZZZZZZZZZZ_L1.htm">ABEJORRAL</option>
   <option value="../99AL/DAL01007ZZZZZZZZZZZZ_L1.htm">ABRIAQUI</option>
   <option value="../99AL/DAL01010ZZZZZZZZZZZZ_L1.htm">ALEJANDRIA</option>
   <option value="../99AL/DAL01013ZZZZZZZZZZZZ_L1.htm">AMAGA</option>
   <option value="../99AL/DAL01300ZZZZZZZZZZZZ_L1.htm">YONDO-CASABE</option>
   <option value="../99AL/DAL01301ZZZZZZZZZZZZ_L1.htm">ZARAGOZA</option>
</select>

Following previous questions on SO such as this one (Selenium - Python - drop-down menu option value), I've first tried the basic way to go about drop-down menus with Selenium, i.e. using its Select method in the following manner:

#Preamble to code
#Importing packages
import selenium
import time
from selenium import webdriver

url = https://elecciones.registraduria.gov.co:81/esc_elec_2015/99AL/DAL01004ZZZZZZZZZZZZ_L1.htm #Setting URL

driver = webdriver.Chrome()
driver.implicitly_wait(5)
driver.get(url)

time.sleep(20)


from selenium.webdriver.support.ui import Select

select = Select(driver.find_element_by_id('combo3')) #Selecting the drop-down menu by ID
select.select_by_value("../99AL/DAL01007ZZZZZZZZZZZZ_L1.htm") # Trying to select the option in drop-down menu by value. Error arises.

I get an ElementNotVisibleException error when selecting by value. So I try a bit of troubleshooting working around other solutions offered on SO (see here: Selecting a value from a drop-down option using selenium python) such as finding the element by its XPath and then clicking it. Here is the code I tried: 

driver.find_element_by_xpath("//select[@id='combo3']/option[@value='../99AL/DAL01007ZZZZZZZZZZZZ_L1.htm']").click()

Here again, an ElementNotVisibleException appears. So next, I try what is suggested here (ElementNotVisibleException: Message: element not visible - Python3 Selenium), i.e. implementing some waiting time and waiting until the element corresponding to the drop-down menu becomes clickable using the following code:

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By

element = WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, "//select[@id='combo3']")))
element.click()

Here I get a TimeoutException. I previously checked using the developer tools console of Chrome if there were two or more elements that matched the locator and there is only one (length: 1) so I think something else may be causing the Timeout but I can't manage to find what it could be.

I also get a TimeoutException when trying some workarounds such as suggested here (Python selenium drop down menu click) by implementing other methods like EC.visibility_of_element_located. Here is the code that raises the TimeoutException:

element = WebDriverWait(driver, 10).until(EC.visibility_of_element_located((By.XPATH, "//select[@id='combo3']")))
select = Select(element)

As you can see, I've tried various techniques to try and select a different option from the Municipio drop-down menu but still can't manage to figure it out. Any help or advice in telling me what I'm missing or doing wrong would be greatly appreciated. Thanks in advance!

Upvotes: 0

Views: 1170

Answers (1)

Matt Shirley
Matt Shirley

Reputation: 4215

The select element with ID 'combo3' has the style "display: none;" which is making it invisible to Selenium. You can drive the element using the div ID 'combo3_chosen' and input the result you want. 

from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By

select_div = WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.XPATH, "//div[@id='combo3_chosen']")))
select_div.click()
input = WebDriverWait(driver, 10).until(EC.visibility_of_any_elements_located((By.XPATH, "//input")))
input[0].send_keys("Todos")
input[0].send_keys(Keys.RETURN)

In my example, I click the select_div which opens an input with a drop-down. I then input 'Todos' and press return to load the 'Todos' section.

I used 'visibility_of_any_elements_located' for the input because there are multiple inputs on the page that are hidden (the other selects). If I use 'visibility_of_element_located' Selenium tries to return the first input which is not visible. 'visibility_of_any_elements_located' will return any visible input.

Alternatively, you could open the select_div and locate the result you want and click that element.

Upvotes: 0

Related Questions