CODEWITHSUNDEEP
pythonseleniumtimeweb-scraping
Jonathan Ishii
Jonathan Ishii

Reputation: 397

Selenium Python - Explicit waits not working

I am unable to get explicit waits to work while waiting for the page to render the js, so I am forced to use time.sleep() in order for the code to work as intended.

I read the docs and still wasn't able to get it to work. http://selenium-python.readthedocs.io/waits.html

The commented out section of code with the time.sleep() works as intended. The WebDriverWait part runs but does not wait.

from selenium import webdriver
import time
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.common.exceptions import TimeoutException

driver = webdriver.Chrome()
url = "https://www.target.com/"

# tells the driver to wait up to 10 seconds before timing out
# for data that will be loaded on the screen
DELAY = 10
driver.implicitly_wait(DELAY)
SLEEP_TIME = 1

# navigate to the page
driver.get(url)
time.sleep(SLEEP_TIME)

try:

    WebDriverWait(driver, DELAY).until(EC.visibility_of_element_located((By.XPATH, """//*[@id="js-toggleLeftNav"]/img"""))).click()
    WebDriverWait(driver, DELAY).until(EC.visibility_of_element_located((By.XPATH, """//*[@id="5"]"""))).click()
    WebDriverWait(driver, DELAY).until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, "#leftNavigation > ul:nth-child(2)")))
    """
    # opens up the side bar javascript
    driver.find_element_by_xpath("""//*[@id="js-toggleLeftNav"]/img""").click()
    time.sleep(SLEEP_TIME)

    # clicks on browse by category
    driver.find_element_by_xpath("""//*[@id="5"]""").click()
    time.sleep(SLEEP_TIME)

    # gets all the category elements
    items = driver.find_element_by_css_selector("#leftNavigation > ul:nth-child(2)").find_elements_by_tag_name("li")
    time.sleep(SLEEP_TIME)
    """
    # gets the hyperlink and category name but the first and the last,
    # since the first is back to main menu and the last is exit
    category_links = {}
    for i in range(1, len(items) - 1):
        hyperlink = items[i].find_element_by_tag_name('a').get_attribute('href')
        category_name = items[i].text
        category_links[category_name] = hyperlink

    print(category_links)

except:
    print("Timed out.")

Upvotes: 2

Views: 3201

Answers (1)

Spence Wetjen
Spence Wetjen

Reputation: 199

This version successfully loads the site, waits for it to render, then opens the side menu. Notice how the wait.until method is is used successfully wait until the page is loaded. You should be able to use the pattern below with the rest of your code to achieve your goal.

CODE

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

driver = webdriver.Chrome()
wait = WebDriverWait(driver, 10)
driver.get("https://www.target.com/")

wait.until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, "#leftNavigation > ul:nth-child(2)")))
button = driver.find_element_by_xpath("""//*[@id="js-toggleLeftNavLg"]""")
button.click()
time.sleep(5)


driver.quit()

Upvotes: 2

Related Questions