CODEWITHSUNDEEP
pythonselenium
floflo29
floflo29

Reputation: 2321

Selenium: different behaviour between driver.implicitly_wait() and time.sleep()

I was trying to scrape a website with Selenium (as it is a website with a dynamically loaded content).

However, to wait for such a dynamic content to be loaded, while I usually use time.sleep(), I have just known about (and tried) driver.implicitly_wait() but it seems I do not get the expected content.

Here is the code:

from selenium import webdriver
import os
import time

os.environ['MOZ_HEADLESS'] = '1'

baseSite = 'https://bair.berkeley.edu/students.html'
driver = webdriver.Firefox()
#driver.implicitly_wait(5) --> full content is not retrieved
driver.get(baseSite)

time.sleep(5) # full content is retrieved

source = driver.page_source
print(source)

Upvotes: 0

Views: 418

Answers (2)

sabraship
sabraship

Reputation: 99

You could use an explicit wait, which would wait until a certain condition is met. For example,

element = WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.ID, "myDynamicElement")))

would wait until the WebDriver located that element (until 10 seconds have passed, at which point, it would time out). This link is helpful for learning more about different types of waits in Selenium. https://selenium-python.readthedocs.io/waits.html

Upvotes: 0

JimEvans
JimEvans

Reputation: 27486

The setting of an implicit wait timeout (i.e., using implicitly_wait() does not affect how the browser loads a page. What that method does is poll the DOM for the desired element when using find_element or find_elements. In the code you posted, setting the implicit wait timeout has no effect because you are not attempting to find any elements on the page.

If you were to provide more details about what you’re expecting (aside from saying, “I want the page to be ‘fully loaded’”, because that phrase is so vague as to be meaningless), it might be easier to provide more guidance.

Upvotes: 1

Related Questions