CODEWITHSUNDEEP
seleniumfirefoxgeckodriver
Kelvin Tan
Kelvin Tan

Reputation: 992

Selenium, Firefox geckodriver does not wait the page to fully load when the Internet is slow

I am using selenium with Firefox driver (geckodriver) to get some page source from a list of urls.

I notice that if the Internet is slow, FireFox does not wait until the page is fully loaded (the execution does not wait at line 5). As a result, the page_source in line 9 is actually from the previous url.

How can I make Firefox to wait for the page to fully load?

Selenium: 3.14.1
Geckodriver: 0.23.0 linux64

1        browser = webdriver.Firefox()
2    
3        for url in url_list:
4          
5            browser.get(url)
6    
7            sleep(1)
8    
9            page_source = browser.page_source
10    
11           if  html == page_source:
12    
13                print "error: page not fully loaded"
14    
15                exit(1)
16           html = page_source

Update: I have tested with Chrome driver. Chrome driver does wait until the page is fully loaded. So maybe the issue is with FireFox driver.

Upvotes: 1

Views: 2899

Answers (3)

Mitul Lakhani
Mitul Lakhani

Reputation: 74

You can wait using WebDriverWait until particular element is located or in case of network is slowed down and also add implicit wait then get page source

Please find below java code:

WebDriverWait wait = new WebDriverWait(driver, 20);

wait.until(ExpectedConditions.stalenessOf(element));

Upvotes: 0

Ashish Kamble
Ashish Kamble

Reputation: 2627

You have to check that ready state is complete for the loaded page in browser or not, till that you can put wait for driver using, t will throw error so you should put this inside of try catch block. 

driver.get(url)
WebDriverWait(driver, 30).until(readystate_complete)

there are different Ready states,like
loading, complete and interactive
usually in javascript people does document.readystate
and the complete means that the document was fully read and all resources (like images) are loaded too

Upvotes: 1

Dhamo
Dhamo

Reputation: 1251

Usually ".get" in Selenium webdriver is done using an HTTP GET operation, and the method will block until the load is complete. So, I don't think slow internet connection causing the issue but there is a chance for the issue to happen if you have interrupted internet connection.

If the page is same, you can leverage Explicit Wait to wait for any element on the page and if the element does not visible or not loaded then you can again hit ".get" method or you can do a ".refresh()" method as you need.

Apparently, if pages are different then wait for the body tag[ xpath: "//body"] to be loaded using the Explicit wait.

For more details about Explicit wait refer here

Upvotes: 1

Related Questions