How to execute tests in a test suite in parallel

Question

I have managed to run tests in parallel by using webdriver against a selenium hub and node. This code called before the tests is run.

cls.driver = webdriver.Remote(
   command_executor="http://localhost:4444/wd/hub",
   desired_capabilities={
        "browserName": "chrome",
        })

    cls.driver.maximize_window()
    cls.driver.get(cls.serverUrl)
    p = multiprocessing.Process(target=cls.driver.get(cls.serverUrl), args=())
    p.start()
    p.join()

In that way I can start multiple browsers by executing them manually from Eclipse. However I would like to do that automatically in a testsuite. But in a test suite all tests are started in a sequence. If anyone has an idea how to proceed it would be great.

Answer 1

Preliminaries

I prepared some sample tests for playing with. These are some simple page title checks. We have one module test_google.py with two unit tests that check the titles of www.google.com and mail.google.com:

# test_google.py

import unittest
from selenium import webdriver


class GoogleTests(unittest.TestCase):

    def setUp(self):
        self.driver = webdriver.Chrome()

    def tearDown(self):
        self.driver.close()


    def test_google_page_title(self):
        self.driver.get('https://www.google.com')
        assert self.driver.title == 'Google'

    def test_gmail_page_title(self):
        self.driver.get('https://mail.google.com')
        assert self.driver.title == 'Gmail'

The second module is test_stackoverflow.py that contains one test that checks the title of stackoverflow.com:

# test_stackoverflow.py

import unittest
from selenium import webdriver


class StackoverflowTests(unittest.TestCase):

    def setUp(self):
        self.driver = webdriver.Chrome()

    def tearDown(self):
        self.driver.close()


    def test_so_page_title(self):
        self.driver.get('https://stackoverflow.com')
        assert 'Stack Overflow' in self.driver.title

Running the tests with the bare unittest runner yields:

$ python setup.py test
running test
running egg_info
...
running build_ext
test_gmail_page_title (test_google.GoogleTests) ... ok
test_google_page_title (test_google.GoogleTests) ... ok
test_so_page_title (test_stackoverflow.StackoverflowTests) ... ok

----------------------------------------------------------------------
Ran 3 tests in 11.657s

OK

Migration to pytest

Install pytest via pip:

$ pip install pytest

pytest supports unit tests out of the box, so we don't need to touch the tests, we can run them immediately. Trying out the pytest runner:

$ pytest -v
================ test session starts ================
platform darwin -- Python 3.6.3, pytest-3.2.5, py-1.5.2, pluggy-0.4.0 -- /Users/hoefling/.virtualenvs/stackoverflow/bin/python3.6
cachedir: .cache
rootdir: /Users/hoefling/projects/private/stackoverflow/so-47439103, inifile:
collected 3 items                                                                                                                                                                                   

test_google.py::GoogleTests::test_gmail_page_title PASSED
test_google.py::GoogleTests::test_google_page_title PASSED
test_stackoverflow.py::StackoverflowTests::test_so_page_title PASSED

================ 3 passed in 13.81 seconds ================

Running tests in parallel

This requires pytest-xdist plugin for pytest. Install it via pip:

$ pip install pytest-xdist

The plugin is installed now, but won't be active by default, so if you run pytest again, you won't notice any difference. Use the numprocesses key to parallelize the test execution. This denotes the number of processes that are reserved to run the tests, here I use the auto value to spawn as many processes as many CPUs my machine has:

$ pytest -v --numprocesses=auto
================ test session starts ================
platform darwin -- Python 3.6.3, pytest-3.2.5, py-1.5.2, pluggy-0.4.0 -- /Users/hoefling/.virtualenvs/stackoverflow/bin/python3.6
cachedir: .cache
rootdir: /Users/hoefling/projects/private/stackoverflow/so-47439103, inifile:
plugins: xdist-1.20.1, forked-0.2
[gw0] darwin Python 3.6.3 cwd: /Users/hoefling/projects/private/stackoverflow/so-47439103
[gw1] darwin Python 3.6.3 cwd: /Users/hoefling/projects/private/stackoverflow/so-47439103
[gw2] darwin Python 3.6.3 cwd: /Users/hoefling/projects/private/stackoverflow/so-47439103
[gw3] darwin Python 3.6.3 cwd: /Users/hoefling/projects/private/stackoverflow/so-47439103
[gw0] Python 3.6.3 (v3.6.3:2c5fed86e0, Oct  3 2017, 00:32:08)  -- [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)]
[gw1] Python 3.6.3 (v3.6.3:2c5fed86e0, Oct  3 2017, 00:32:08)  -- [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)]
[gw2] Python 3.6.3 (v3.6.3:2c5fed86e0, Oct  3 2017, 00:32:08)  -- [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)]
[gw3] Python 3.6.3 (v3.6.3:2c5fed86e0, Oct  3 2017, 00:32:08)  -- [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)]
gw0 [3] / gw1 [3] / gw2 [3] / gw3 [3]
scheduling tests via LoadScheduling

test_google.py::GoogleTests::test_google_page_title 
test_stackoverflow.py::StackoverflowTests::test_so_page_title 
test_google.py::GoogleTests::test_gmail_page_title 
[gw0] PASSED test_google.py::GoogleTests::test_gmail_page_title 
[gw1] PASSED test_google.py::GoogleTests::test_google_page_title 
[gw2] PASSED test_stackoverflow.py::StackoverflowTests::test_so_page_title 

================ 3 passed in 7.81 seconds ================

You will see that all the three tests run in parallel by three chrome instances opened simultaneously. Each test runs in own process, so they don't interfere with each other. Also, notice that both test methods from the GoogleTests class also run in parallel, so this is not restricted to tests in different modules or classes.

Integration with setup.py

When I first started the migration to pytest, one of the conditions I had was that the command python setup.py test should still work so we don't need to memorize an extra pytest command to run the tests and so we also don't have to adapt all our utility scripts or build jobs on our integration server, so here are the steps to update your setup.py script:

1. Add the following packages to test requirements:

   from setuptools import setup
   
   setup(
       ...
       tests_require=[
           'pytest',
           'pytest-runner',  # this one is needed to install distutils command for pytest
           'pytest-xdist'
       ],
   )
   

2. Add an alias to setup.cfg:

   [aliases]
   test=pytest
   

3. Add a configuration section for pytest in setup.cfg:

   [tool:pytest]
   addopts=--verbose --numprocesses=auto
   

Now, if you run python setup.py test, the correct runner will be invoked and the xdist plugin will be active by default.

Additional notes

I personally really like pytest as it offers much more than a plain test execution - you can write tests as pure functions (no wrapping into TestCase classes required), collect tests without executing them, easily rerun only tests that failed recently, hook the code coverage measurement with multiple reports in different formats and many more. Refer to the official docs for more details, it is really worth the reading time!