CODEWITHSUNDEEP
pythonseleniumweb-crawler
FranGoitia
FranGoitia

Reputation: 1993

Selenium takes too much time loading a page

I'm trying to crawl whoscored which has a lot of stuff and takes a lot of time to load completely. However, the stuff that I want are loaded pretty fast, but nonetheless driver.get(url) executes until the webpage is loaded completely. Is there any way to prevent this from happenning ? To make the get method return as soon as some element I define is present in the DOM ? I'm thinking something as stop works in a browser.

Upvotes: 0

Views: 3431

Answers (1)

FranGoitia
FranGoitia

Reputation: 1993

So, the best solution I found for the kind of behaviour I wanted was:

profile = webdriver.FirefoxProfile()
profile.set_preference("webdriver.load.strategy", "unstable")
profile.update_preferences()
driver = webdriver.Firefox(firefox_profile=profile)
driver.get(url)
try:
    wait = WebDriverWait(driver, timeout=20, poll_frequency=0.1)
    wait.until(<expectation object>)
finally:
    driver.execute_script("return window.stop")

This will stop the browser loading the page and you can still crawl and interact with the loaded site.

Upvotes: 2

Related Questions