CODEWITHSUNDEEP
pythonselenium
DSellers
DSellers

Reputation: 23

How to periodically re-check a webpage using selenium in python

I am new to selenium in python (and all web-interface applications of python) and I have a task to complete for my present internship.

My script successfully navigates to an online database and inputs information from my data tables, but then the webpage in question takes anywhere from 30 seconds to several minutes to compute an output.

How do I go about instructing python to re-check the page every 30 seconds until the output appears so that I can parse it for the data I need? For instance, which functions might be I start with?

This will be part of a loop repeated for over 200 entries, and hundreds more if I am successful so it is worth my time to automate it.

Thanks

Upvotes: 0

Views: 1039

Answers (2)

bkemmer
bkemmer

Reputation: 420

You should use Seleniums Waits as pointed by G_M and Sam Holloway. One which I most use is the expected_conditions:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

driver = webdriver.Firefox()
driver.get("http://somedomain/url_that_delays_loading")
try:
    element = WebDriverWait(driver, 10).until(
        EC.presence_of_element_located((By.ID, "myDynamicElement"))
    )
finally:
    driver.quit()

It will wait until there is an element with id "myDynamicElement" and then execute the try block, which should contain the rest of your work. I prefer to use the the class By.XPATH, but if you use By.XPATH with the method presence_of_element_located add another () so it will be the required tuple as noted in this answer:

from selenium.webdriver.common.by import By

driver.find_element(By.XPATH, '//button[contains(text(),"Some text")]')
driver.find_element(By.XPATH, '//div[@id="id1"]')
driver.find_elements(By.XPATH, '//a')

The easiest way to find (for me) the XPATH of an element is going to the developer mode in chrome (F12), pressing ctrl+F, and using the mouse with inspect, trying to compose the proper XPATH, which will be specific enough to find just the expected element, or the least number of elements as possible.

All the examples are from (or based) the great selenium documentation.

Upvotes: 1

Sam Holloway
Sam Holloway

Reputation: 121

If you just want to space out checks, the time.sleep() function should work.

However, as G_M's comment says, you should look into Selenium waits. Think about this: is there an element on the page that will indicate that the result is loaded? If so, use a Selenium wait on that element to make sure your program is only pausing until the result is loaded and not wasting any time afterwards.

Upvotes: 1

Related Questions