Is there any way to give limit time for Selenium to find element/s?

Question

I am trying to automate Google Maps using Python Selenium Chrome driver, my code is stable to get what I need, but not the performance time.

I use iterative code to get merchants data over Google Maps, the time rate of getting 1 merchant data is below 1 second using Selenium, but when the code caught an exception, it will take very long like 6-7 seconds.

Here is some summary of execution time:

https://picoolio.net/image/dgxC (Normal Execution Time)

https://picoolio.net/image/dgxT (Execution Time when exception occured on Open Hour, too long)

https://picoolio.net/image/dgxV (Execution Time when exception occured, sometimes run as Expected)

Here is some code to get open hour list:

try:
      openhour = wait(driver, 1).until(EC.visibility_of_element_located((By.XPATH, 
                           './/span[@class="section-open-hours-button maps-sprite-pane-info-arrowup"]')))

      driver.execute_script("arguments[0].scrollIntoView(true);", openhour)
      openhour.click()
      openhourstacked = wait(driver, 1).until(EC.visibility_of_element_located((By.CLASS_NAME, 
                                  'section-open-hours-container-hoverable'))).get_attribute("aria-label")
      openhourlist = openhourstacked.split(",")
      openhourlist[len(openhourlist) - 1] = openhourlist[len(openhourlist) - 1].split(".")
      openhourlist[len(openhourlist) - 1] = openhourlist[len(openhourlist) - 1][0]

      except NoSuchElementException:
         openhourlist = []
         print("No Open Hour list in this merchant!")
         openhour_trig = True
      except WebDriverException:
         openhourlist = []
         print("Failed to load Open Hour list in this merchant!")
         openhour_trig = True

Lots of recommendations say that to do an explicit wait correctly, I should use a combination of WebDriverWait and EC (Expected Conditions), and I tried it directly with the above code, but the WebDriverWait seems not to work for all of the incoming exceptions, in other words sometimes succeeds and sometimes fails.

I tried to replace EC.visibility_of_all_element_located with EC.presence_of_all_element_located, but it doesn't make any difference.

I expected to make the execution time still smooth when there is any exception occured, is there any way to make Selenium stop finding elements in case of a timeout, besides the above way? Or is this an internet connection issue? Any thoughts welcome!

Update: So I modify open hour code to something like this

try:
              if len(driver.find_elements(By.XPATH, './/span[@class="section-open-hours-button maps-sprite-pane-info-arrowup"]'))>0:
                        openhour = wait(driver, 5).until(EC.visibility_of_element_located((By.XPATH, 
                                       './/span[@class="section-open-hours-button maps-sprite-pane-info-arrowup"]')))
                        #openhour = driver.find_element_by_xpath('.//span[@class="section-open-hours-button maps-sprite-pane-info-arrowup"]')
                        driver.execute_script("arguments[0].scrollIntoView(true);", openhour)
                        openhour.click()
                        #openhourstacked = driver.find_element_by_class_name('section-open-hours-container-hoverable').get_attribute("aria-label")
                        openhourstacked = wait(driver, 5).until(EC.visibility_of_element_located((By.CLASS_NAME, 
                                              'section-open-hours-container-hoverable'))).get_attribute("aria-label")
                        openhourlist = openhourstacked.split(",")
                        openhourlist[len(openhourlist) - 1] = openhourlist[len(openhourlist) - 1].split(".")
                        openhourlist[len(openhourlist) - 1] = openhourlist[len(openhourlist) - 1][0]
                    else:
                        openhourlist = []
                        print("No Open Hour list in this merchant! (Not an Exception)")
                except NoSuchElementException:
                    openhourlist = []
                    print("No Open Hour list in this merchant!")
                    openhour_trig = True
                except WebDriverException:
                    openhourlist = []
                    print("Failed to load Open Hour list in this merchant!")
                    openhour_trig = True

Yeah, this way it really avoid Exception, But this repeats the search like before, so it might give a long search up to 6-7 seconds again when the elements not found.. explicit wait too doesn't really helpful in these terms i think

Answer 1

I know that this is already long question and get the solutions, Forgot to update again, but I think, I realized that setting implicit wait to several second, eg: 2 seconds like this driver.implicitly_wait(2) will help to limit the WebDriver searching to that last seconds, over than the last seconds, It will throw the Exception

Answer 2

Summary:

In your code try to increase the time of the explicit wait. It won't slow you down but it'll prevent those exceptions. Try 5 seconds instead of 1 second. Try this wait(driver,5)

---

We use waits because it takes time to load DOMs and sometimes we need to wait for page/elements to load/hide/change and we have to act according to the change.
What explicit waits does is it waits up to a given time for the expected condition to be fulfilled and checks every 500miliseconds that if the conditions are fulfilled or not. This way you lose max 500miliseconds time before executing the next task. If the conditions are not fulfilled in time it will throw a timeout exception. Explicit waits will not solve all your problems.

You still need to check for possible exceptions and decide what to do when that occurs. Some sites are slow to load or your bandwidth is low sometimes. Then you have no way to speed the process up.

You have to first analyze the page and make sure what to expect. If you are waiting for an element to be present but there is a possibility that it might not, then you should handle the exception accordingly.

If you are checking for an element exists in a page or not then it would be better to not wait for the element but try to use find_elements_ and check the size of the list. This way you would avoid exceptions and it would be faster.

driver.find_elements(By.ID, "locator").size()>0

Answer 3

yes explicit wait (webdriver wait ) resolves this .. Else simple way is to create a for loop with timeseconds and write the find elements inside the loop, so even it fails it will check for 180secs and write conditions beyond the expected wait time

  for (int second = 0; second < 60; second++) {
  try {} 
  catch (WebDriverException we) 
  {}
  if (second >= 60) {
  //write failure 
  }