CODEWITHSUNDEEP
pythontestingpytest
guettli
guettli

Reputation: 27995

Run only tests which depend on the change

How to detect which tests pytest need to run?

Up to now I run all tests in CI. This is slow.

Isn't there a way to automatically detect which tests need to run, and only execute a fraction of the whole test suite?

Background: I am reading the Google SE book and read about their gigantic monorepo and that their tool Blaze can detect which tests need to run. Sounds very cool.

Upvotes: 12

Views: 3499

Answers (3)

hoefling
hoefling

Reputation: 66451

I had good experience with pytest-testmon which determines the subset of tests to execute on code change based on test coverage. The details on that are explained well in Determining affected tests section of testmon's docs, but the general advice is to maintain high line/branch coverage for testmon to be effective. This also means that it will fail where coverage fails.

If you want to use it in the CI, you need to cache the .testmondata database file between test jobs and ensure the --testmon arg is used when invoking pytest. It also depends on your CI provider, but most popular ones out there offer caching mechanisms (Github, Gitlab, CircleCI, Appveyor etc).

testmon can also be effectively combined with pytest-watch watchdog to keep a daemon for test execution running. An example command that reruns subset of tests on code changes and sends system notifications on success or failure:

$ pytest-watch --runner="pytest --testmon" \
  --onfail="notify-send --urgency=low -i error 'Tests Failed'" \
  --onpass="notify-send --urgency=low -i info 'Tests passed'"

You can also use Bazel for building and testing as well; however, IMO it is an overkill for a pure Python project and is not worth it for just the incremental testing purposes.

Upvotes: 9

mdmjsh
mdmjsh

Reputation: 945

You can use the pytest-incremental plugin for this which uses AST to analyse the import structure of a project and only runs tests for which the source code has changed.

pip install pytest-incremental
python -m pytest --inc

enter image description here

Analyzing the graph is easy to see that a change in app would cause only test_app to be execute. And a change in util would cause all tests to be executed.

from the docs

Upvotes: 6

Guy
Guy

Reputation: 50909

You can add @pytest.mark to the tests

@pytest.mark.SomeFeature
@pytest.mark.Regression
def test_example1(self):
    pass

@pytest.mark.AnotherFeature
@pytest.mark.Regression
def test_example2(self):
    pass

And use it when triggering the tests

pytest TestsFolder -m SomeFeature

This will run only test_example1, -m Regression will run both tests.

Upvotes: 3

Related Questions