How to get a new page after selenium times out on indefinitely loading page

Question

I've come across a problematic page that causes Selenium Chrome (selenium version 3.10.0 in python 3, chromedriver Version 2.35.528157) on MacOSX to time out, I think because there is something indefinitely loading on the page. The problem is that after that timeout, all future requests to the driver to .get() a new url also fail with a timeout, even if they worked before. In fact, observing the browser it is never sent to the new url. This, of course, renders the browser useless for further sessions.

How can I "reset" the driver so that I can carry on using it? Or failing that, how can I debug why the .get() command doesn't seem to work after visiting the problematic page. The code and my output are below (problematic page is http://coastalpathogens.wordpress.com/2012/11/25/onezoom/: I'd be interested if other people see the same thing, and with other pages too

from selenium import webdriver
from selenium.common.exceptions import TimeoutException

browser = webdriver.Chrome()
browser.set_page_load_timeout(10)
browser.implicitly_wait(1)

for link in ("http://www.google.com", "http://coastalpathogens.wordpress.com/2012/11/25/onezoom/","http://www.google.com"):
    try:
        print("getting {}".format(link))
        browser.get(link)
        print("done!")
    except TimeoutException:
        print("Timed out")
        continue

result:

getting http://www.google.com
done!
getting http://coastalpathogens.wordpress.com/2012/11/25/onezoom/
Timed out
getting http://www.google.com
Timed out

Answer 1

As per your question and your own code block I have executed your own code tweaking a few ChromeDriver settings through the chrome.options class as below and it works perfecto :

Code Block :

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.common.exceptions import TimeoutException

options = Options()

options.add_argument("start-maximized")
options.add_argument("disable-infobars")
options.add_argument("--disable-extensions")
browser = webdriver.Chrome(chrome_options=options, executable_path=r'C:\Utility\BrowserDrivers\chromedriver.exe')
browser.set_page_load_timeout(10)

for link in ("http://www.google.com", "http://coastalpathogens.wordpress.com/2012/11/25/onezoom/","http://www.google.com"):
    try:
        print("getting {}".format(link))
        browser.get(link)
        print("done!")
    except TimeoutException:
        print("Timed out")
    continue

Console Output :

getting http://www.google.com
done!
getting http://coastalpathogens.wordpress.com/2012/11/25/onezoom/
done!
getting http://www.google.com
done!

Issue at your end and the solution

There are a couple of things which you need to consider as follows :

• Until and unless your usecase have a constraint on Page Load Timeout you must not use set_page_load_timeout() as on slow networks, while invoking urls e.g. http://coastalpathogens.wordpress.com/2012/11/25/onezoom/ the Browser Client may require more then 10 seconds (i.e. the configured time through set_page_load_timeout(10)) to send document.readyState equal to "complete" to Selenium.

• If your usecase have a dependency on Page Load Timeout, catch the exception and invoke quit() to shutdown gracefully as follows :

  from selenium import webdriver
  
  driver = webdriver.Chrome(executable_path=r'C:\path\to\chromedriver.exe')
  driver.set_page_load_timeout(2)
  try :
      driver.get("https://www.booking.com/hotel/in/the-taj-mahal-palace-tower.html?label=gen173nr-1FCAEoggJCAlhYSDNiBW5vcmVmaGyIAQGYATG4AQbIAQzYAQHoAQH4AQKSAgF5qAID;sid=338ad58d8e83c71e6aa78c67a2996616;dest_id=-2092174;dest_type=city;dist=0;group_adults=2;hip_dst=1;hpos=1;room1=A%2CA;sb_price_type=total;srfid=ccd41231d2f37b82d695970f081412152a59586aX1;srpvid=c71751e539ea01ce;type=total;ucfs=1&#hotelTmpl")
      print("URL successfully Accessed")
      driver.quit()
  except :
      print("Page load Timeout Occured. Quiting !!!")
      driver.quit()
  

• Console Output :

  Page load Timeout Occured. Quiting !!!
  

  You can find a detailed discussion on set_page_load_timeout() in How to set the timeout of 'driver.get' for python selenium 3.8.0?

• Consider replacing the usage of implicitly_wait() by ExplicitWait. Modern websites uses JavaScript, Ajax Calls and React Native where WebDriverWait will come into play and you can't mix up implicitly_wait() with WebDriverWait().