Selenium crashes after running around 500 tests

Question

I am trying to run some dynamically generated tests. They work perfectly up to around 500, then I just get the error below. Has anyone seen this before?

ChromeDriver executable needs to be available in the path.  

I was thinking that it might be a machine spec issue. I have a hyper-threaded i5 and 8GB of RAM. Watching the system monitor, I do not see memory get above 6GB and CPU never reaches 100% on any core. I am running Linux Mint.

I have tried adding timeouts after the browser has closed but it doesn't seem to do anything. I have noticed that there are loads of the ChromeDriver processes. Is their some kind of limit on this process?

Any help would be appreciated.

Answer 1

Short answer

EDIT: Confirmed, two bugs with fixes in master here and here; released in 2.46.0. The following is relevant to 2.45 and earlier.

It appears that there is a bug in nose and/or selenium. I can replicate this bug when running tests with nose; I suspect there are other ways to trigger it.

If you need a solution before the bug fixes land, you can do one of the following:

• Increase the max # of open files permitted ... OR ...
• ... tweak your site-packages/selenium/webdriver/chrome/service.py file

I'm guessing that the output of ulimit -n is 1024 on your machine. Why do I think this? 500 * 2 is close to 1024 (and it's a common setting) ... read on.

---

Long answer

This is a subtle bug in that only happens when you run tests in a certain way, with certain limits in the environment.

Making this hard to identify is a selenium bug that results in an incorrect error message.

Bug #1 ... Too broad an exception

Selenium is hiding the REAL error from you by using too broad an exception. If your test passes for the first 122 runs (as mine did) and then starts failing for every subsequent test with ChromeDriver executable needs to be available in the path ... well something weird is going on. You OBVIOUSLY had chromedriver in the correct path for the first 122 (or 500) tests.

So let's get the real error message by telling selenium to stop assuming every raised exception is one in which the chromedriver binary isn't in the PATH environment variable.

Change the start method in site-packages/selenium/webdriver/chrome/service.py

From:

try:
    self.process = subprocess.Popen([
      self.path,
      "--port=%d" % self.port] +
      self.service_args, env=env, stdout=PIPE, stderr=PIPE)
except:
    raise WebDriverException(
        "'" + os.path.basename(self.path) + "' executable needs to be \
        available in the path. Please look at \
        http://docs.seleniumhq.org/download/#thirdPartyDrivers \
        and read up at \
        http://code.google.com/p/selenium/wiki/ChromeDriver")

To:

try:
    self.process = subprocess.Popen([
      self.path,
      "--port=%d" % self.port] +
      self.service_args, env=env, stdout=PIPE, stderr=PIPE)
except:
    # let all exceptions reach the user, with error type and message
    # for demonstration purposes only
    raise  

Now, re-run your 500 tests. You will get a much more helpful error message, probably: OSError: [Errno 24] Too many open files

Bug #2 ... Too many open files

For some reason, stdout and stderr are not closing.

I can force this bug with the following test file duplicated enough times.

# bash to duplicate: 
# for i in `seq 1 130`
# do
#    cp test_std_close.py test_std_close_$(printf %03d ${i}).py
# done

import unittest 
from selenium import webdriver

class TestStdClose(unittest.TestCase):

    def test_std_close(self):
        driver = webdriver.Chrome();
        driver.get('https://google.com');
        driver.close()

If I put 130 of these (test_001.py, test_002.py, etc) into a directory and run nosetests std_test/test*py

I get a failure on #122, just like my normal test suite. You may need to run it 500 times to duplicate your error.

SOLUTION

The work-around is to either bump up the max # of open files, or to change the stop method in site-packages/selenium/webdriver/chrome/service.py

From:

try:
    if self.process:
        self.process.kill()
        self.process.wait()
except OSError:
    # kill may not be available under windows environment
    pass

To:

try:
    if self.process:
        self.process.stdout.close()  # add this line
        self.process.stderr.close()  # and this one
        self.process.kill()
        self.process.wait()
except OSError:
    # kill may not be available under windows environment
    pass

Either of these hacks will ensure my 160 tests all execute on a variety of Mac OS configurations (10.6, 10.9, 10.10).

I am in the process of submitting bug reports and patches to selenium; the root of the issue may be somewhere else, but one of these two changes fixed my test suite (run via nose).

Conclusion

So, why did this fail after ~500 for you? I'm guessing ulimit -n reports 1024 on your machine. Some math: 500 * 2 (stdout, stderr) == 1000, leaving you 24 files to play with.

Why did this fail after ~122 for me? Because on my Mavericks MacBook Pro ulimit -n reports 256. Some math: 122 * 2 == 244, leaving me 12 open files to play with.

Answer 2

Found some suggestion here saying: (for Linux) place chromedriver under /usr/bin. For windows, please have the chromedriver placed under /Python27/Scripts

However, my experience on Win 7 is put the driver.exe files, i.e. chromedriver.exe, IEDriverServer.exe in the /Python27/ directly works.

Answer 3

The solution I have found is not the best but it does seem to work well enough for the purposes that I am using it for. What I did was switch the driver I was using from chromedriver to Firefox driver and all the tests now execute without any issue using firefox.

Answer 4

As Karna observed this sounds like previous tests driver instances not being cleaned up.

Are you currently tearing down the browser via:

driver.close()

by any chance?

If so try:

driver.quit()

instead as this will both close the browser window and quit the driver. If you are already doing this then I'm not sure why the driver is staying around :(