VSCode pytest test discovery fails

Question

Pytest test discovery is failing. The UI states:

Test discovery error, please check the configuration settings for the tests

The output window states:

Test Discovery failed: 
Error: Traceback (most recent call last):
  File "C:\Users\mikep\.vscode\extensions\ms-python.python-2019.4.11987\pythonFiles\testing_tools\run_adapter.py", line 16, in <module>
    main(tool, cmd, subargs, toolargs)
  File "C:\Users\mikep\.vscode\extensions\ms-python.python-2019.4.11987\pythonFiles\testing_tools\adapter\__main__.py", line 90, in main
    parents, result = run(toolargs, **subargs)
  File "C:\Users\mikep\.vscode\extensions\ms-python.python-2019.4.11987\pythonFiles\testing_tools\adapter\pytest.py", line 43, in discover
    raise Exception('pytest discovery failed (exit code {})'.format(ec))
Exception: pytest discovery failed (exit code 3)

Here are my settings:

{
    "python.pythonPath": ".venv\\Scripts\\python.exe",
    "python.testing.pyTestArgs": [
        "tests"
    ],
    "python.testing.unittestEnabled": false,
    "python.testing.nosetestsEnabled": false,
    "python.testing.pyTestEnabled": true
}

I can run pytest from the command line successfully FWIW.

Answer 1

Check for randomness.

One of the possibilities not yet discussed here is that the collection step and execution step can disagree if you're using randomness in your parameterization. For example, something like this:

@pytest.mark.parametrize("seed", [random.randint(0, 1000000) for _ in range(20)])

But when you try to run it in VSCode, you get a bunch of:

ERROR: not found: /workspaces/mygame/tests/test_game.py::test_e2e[363440]
(no name '/workspaces/mygame/tests/test_game.py::test_e2e[363440]' in any of [<Module test_game.py>])

ERROR: not found: /workspaces/mygame/tests/test_game.py::test_e2e[241530]
(no name '/workspaces/mygame/tests/test_game.py::test_e2e[241530]' in any of [<Module test_game.py>])

ERROR: not found: /workspaces/mygame/tests/test_game.py::test_e2e[365218]
(no name '/workspaces/mygame/tests/test_game.py::test_e2e[365218]' in any of [<Module test_game.py>])

I have not found a workaround that preserves the randomness at the pytest level, but switching to something hardcoded like this can fix it:

@pytest.mark.parametrize("seed", [360392,992691,571190,204291,...])

So, if you want randomness in your tests, you might have to do it inside the test itself, and leave the parameterization consistent from run to run.

Answer 2

I crushed into a similar issue. First the tests are all shown under the test-icon.

This is my setup in settings.json, not working:

"python.testing.pytestArgs": [
    "--rootdir", "${workspaceFolder}/tests/tests_unit",
    "--html=${workspaceFolder}/build/test_report.html",
    "--cov-report", "term",
    "--cov-report", "html:${workspaceFolder}/build/coverage.html",
    "--cov-branch",
    "--cov=${workspaceFolder}"
],

My problem, and maybe yours too, lay in the line

 "--cov-report", "html:${workspaceFolder}/build/coverage.html",

The file or directory coverage.html is not found, when I click on the "Run all tests"-icon.

When I run the command-line

python -m pytest --html=./build/test_report.html --cov-report term --cov-report html:./build/coverage.html --cov-branch --cov=.

Everything works fine. However removing the stated line from the configuration leads to the tests to shown up again withing the "Test" view, but no html coverage report is generated.

What bothers me is that, the command line created a folder "coverage.html" below build-subdirectory and even when I copy the URL that is given as "not found"-error and paste it into explorer for example, the folder opens up.

So what is expected to be found by that line? A single file? Why is it not created by the run? The folder "assets" below build is created anyway, so it seems to work partly at least.

Answer 3

tldr (checklist)

1. Disable test coverage module
2. Set --rootdir parameter in "python.testing.pytestArgs"
3. Ensure the right VSCode's interpreter is set
4. See if your tests are discoverable (i.e. without errors) with pytest --collect-only

Disable test coverage module

In my case the problem with vscode being unable to discover tests was the coverage module being enabled in the setup.cfg (alternatively this could be a pytest.ini), i.e.

addopts= --cov <path> -ra

which caused the test discovery to fail due to low coverage. The solution was to remove that line from the config file.

Also, as suggested in the documentation:

You can also configure testing manually by setting one and only one of the following settings to true: python.testing.unittestEnabled, python.testing.pytestEnabled, and python.testing.nosetestsEnabled.

In settings.json you can also disable coverage using --no-cov flag:

"python.testing.pytestArgs": ["--no-cov"],

Set --rootdir parameter in "python.testing.pytestArgs"

Slightly related - in case of more complex projects it might be also necessary to change the rootdir parameter (inside your settings.json) when running tests with pytest (in case of ModuleNotFoundError):

    "python.testing.pytestArgs": [
        "--rootdir","${workspaceFolder}/<path-to-directory-with-tests>"
    ],

Ensure the right VSCode's interpreter is set

It could be that VSCode is using a different Python interpreter than your terminal. VSCode's interpreter might not have the right dependencies installed which causes the test discovery to fail.

solution:

Open Command Palette (Cmd + Shift + P) -> "Select interpreter"

Answer 4

I had the same issue. I could solve it by deleting the cash folder .pytest_cache

Answer 5

In my case it was pytestArgs in settings.json set to a specific subdirectory (subdir1) and not detecting others:

{
    "python.testing.pytestArgs": [
        "subdir1"
    ],
    "python.testing.unittestEnabled": false,
    "python.testing.pytestEnabled": true
}

Solution is: either add the subdirs containing tests or empty pytestArgs and refresh

Answer 6

In your case, VSCode is not able to find the needed dependencies it requires to run the tests and do the needed configuration. This is happening because you have your tests configured to be ran inside a virtual environment, but VSCode has no clue about that.

To fix this, you need to instruct VSCode about the paths of your environment so it can do the detection based on that. Paste the following snippet into your .vscode/settings.json file.

{
    "python.testing.pytestEnabled": true,
    "python.testing.pytestPath": "venv/bin/pytest",
    "python.envFile": "${workspaceFolder}/.env",
    "python.testing.pytestArgs": [
        "--no-cov"
    ],
    "python.defaultInterpreterPath": "venv/bin/python"
}

Answer 7

In my case I needed to pip install the a library, that was not installed.

I thought that it will give an error while running not error to show the test in the GUI.

Hope it helps anyone.

Answer 8

Just sharing this since it worked for me.

pytest --collect-only helped me to show the import errors, then run pip install -e . after

Answer 9

I've had headaches with this as well. Got it running now, and I thought I'd share the steps I undertook to get it running.

(Windows 10, Python 3.11, VS Code 1.75.1, pytest 7.2.1)

I don't understand all the steps, and don't know if they are all necessary, but at least it worked.

My project structure is like this:

src
├───.vscode
│       settings.json
│
├───app
│       __init__.py
│       myapp.py
│
└───tests
        __init__.py
        conftest.py
        test_myapp.py

The main folder that I use/open in VS Code is src. Anything above that (docs, venv-related etc.) is not accessible within VS Code with this layout.

Steps:

1. Uninstall and reinstall Python extension in VS Code, but this might not be necessary.

2. Delete all these folders in and under src (recursively), if they exist: .pytest_cache and __pycache__ (although the latter might not be necessary).

3. Put empty __init__.py files in the folders as depicted in the project structure above.

4. src/.vscode/settings.json contains this:

{
    "python.testing.pytestArgs": [
        "tests"
    ],
    "python.testing.unittestEnabled": false,
    "python.testing.pytestEnabled": true
}

5. In the testcase files (including conftest.py) I incude the following, before importing my app (e.g. import app or from app import ... etc.):

from pathlib import Path
import os
import sys

main_folder = Path(__file__).parent.parent
sys.path.insert(0, str(main_folder))
sys.path.insert(0, str(main_folder / 'app'))
sys.path.insert(0, str(main_folder / 'tests'))
os.chdir(main_folder / 'app')

import app

This made the tests discoverable and work correctly inside VS Code.

To run the tests on a standalone basis, I include this at the end of each test file:

@report_execution_time
def main_test():

    os.system('python -m pytest ..')


if __name__ == "__main__":

    main_test()

I use the main_test function, rather than putting its "body" under
if __name__ == "__main__":, so that I can use a decorator on it, in this case to report execution time.

I hope this may help someone or anyone.

Answer 10

There are several causes for this.

The simple answer is, it is not perfect. Furthermore, Python is not native to VS Code; Python extensions may not be playing nice either.

I have observed some mitigating measures, though.

NB VS Code didn't "design" test discovery; that is the job of the testing framework. Make sure you have a basic grasp on how it works.

Troubleshooting

For every "battery" of tests, VS Code will gladly tell you what went wrong, right in the Test view panel. To see this, expand the bar with the name of the project, and hover your mouse over the line that is revealed. Look at the traceback.

In the image above, the error is "No module named src". This is more interesting than it sounds. If I import a file from another file, that import path may not be visible to the test discovery mechanism. It may be running from a different "cwd". The best you can do is try to figure out the topmost path of your project, and either add or remove path qualifiers. Until it works.

Main causes

1. From time to time, VS Code will lose info on the project test configuration. Use command window (Ctrl + Shift + P) to either:

   1. (re)scan tests
   2. (re)configure test spec for the project <-- important!
   3. restart Python Language Server

2. Python is supposed to remove the need for the empty __init__.py; it seems VS Code loves those. They should be in each folder that leads to a test folder. Not every folder, but the top-down path.

3. Depending on what you select as the "root test folder" in the test configuration, it might have different meaning based on what VS Code thinks is the root of your project. That probably goes for any specific folder too.

4. import. VS Code doesn't like syntax errors. It is possible some syntax errors will not get highlighted in the code. This (unfortunately) goes for all the imports in your file. But you shouldn't test invalid code anyway, right?

Minor buggy behaviors

1. Running some other visible test might help refresh the discovery process.
2. VS Code should automatically (by setting) refresh tests on each Save operation. But it doesn't hurt to refresh it manually.

TLDR

1. Look at the error items in the test panel
2. Re-configure project test discovery parameters from time to time
3. Make sure you don't have syntax errors (visible or not)
4. Create empty __init__.pys in each folder of your project that leads to the test folder
5. Clean up your import logic

P.S. The Test view has been extensively worked on and improved much over the course of 1 year. Expect changes in behavior.

Answer 11

Test files need to be named test_*.py or *_test.py for pytest collection to work.

Then run pytest --collect-only at the command line to make sure all of the tests are found. Then magically, the flask icon in VSCode suddenly shows the test files and their tests.

As a noob, I was putting my unit tests inside the module files, since I was following the pattern of unittest, and then running pytest *.py. That doesn't work with pytest, and I didn't find a command line argument override the naming convention.

Answer 12

In my case, I had to make sure that I was using the right interpreter where the libraries where installed (I used a virtual environment) but the interpreter was pointing to the globally installed python. After changing the interpreter to that of the virtual environment, it worked.

Answer 13

Here is generic way to get Django tests to run with full vscode support

1. Configure python tests
   1. Choose unittest
   2. Root Directory
   3. test*.py
2. Then each test case will need to look like the following:

from django.test import TestCase

class views(TestCase):

    @classmethod
    def setUpClass(cls):
        import django
        django.setup()

    def test_something(self,):
        from user.model import something
        ...

Any functions you want to import have to be imported inside the test case (like shown). The setUpClass runs before the test class is setup and will setup your django project. Once it's setup you can import functions inside the test methods. If you try to import models/views at the top of your script, it will raise an exception since django isn't setup. If you have any other preinitialization that needs to run for your django project to work, run it inside setUpClass. Pytest might work the same way, I haven't tested it.

Answer 14

2021-12-22

Please find below the settings I used to get pytest working in VsCode after much frustration. I found many helpful pieces of advice here and elsewhere on the internet, but none were complete enough to spare me a bit of cursing. I hope the following will help someone out. This setup allows me to run testing visually from the test explorer extension and also from the integrated terminal. I am using a src format in my workspace and Conda for environment management. The settings related to the terminal setup keep me from having to manually enable my Conda environment or set the python path. Possibly people who have been using VSCODE for more than two days could add something nice to make this better and/or more complete.

##### Vscode info:

Version: 1.63.2 (Universal) Commit: 899d46d82c4c95423fb7e10e68eba52050e30ba3 Date: 2021-12-15T09:37:28.172Z (1 wk ago) Electron: 13.5.2 Chromium: 91.0.4472.164 Node.js: 14.16.0 V8: 9.1.269.39-electron.0 OS: Darwin x64 20.6.0

##### Testing Extension:

Python Test Explorer for Visual Studio Code extension installed, v. 0.7.0

##### Pytest version

pytest 6.2.5

##### Directory Structure:

     workspace
       .env
       ./src
            __init__.py
            code1.py
            code2.py
       ./tests
            __init__.py
            test_code1.py
            test_code2.py

##### .env file (in root, but see the "python.envFile" setting in settings.json)

PYTHONPATH=src

#####. settings.json

    {
        "workbench.colorTheme": "Visual Studio Dark",
        "editor.fontFamily": " monospace, Menlo, Monaco, Courier New",
        "python.testing.unittestEnabled": false,
        "python.testing.cwd": ".",
        "terminal.integrated.inheritEnv": true,
        "python.envFile": "${workspaceFolder}/.env",
        "python.defaultInterpreterPath": 
        "~/anaconda3/envs/mycurrentenv/bin/python",
        "pythonTestExplorer.testFramework": "pytest",
        "python.testing.pytestEnabled": true,
        "python.testing.pytestArgs": [
           "tests"
        ],
        "python.terminal.activateEnvironment": true,
        "python.terminal.activateEnvInCurrentTerminal": true,
        "terminal.integrated.env.osx": {
            "PYTHONPATH": "${workspaceFolder}/src:${env:PYTHONPATH}"
          }
    }

Answer 15

In my case, it was the vscode python extension problem. I switched the test platform from pytest and switched to it back again, and the tests got discovered. It seems that when testing is universally enabled for all python projects and it fails to discover tests at the beginning, it fails forever!

Answer 16

I spent ages trying to decipher this unhelpful error after creating a test that had import errors. Verify that your test suite can actually be executed before doing any deeper troubleshooting.

pytest --collect-only is your friend.

Answer 17

In my case, same problem appeared each time when flake8 linter reported errors. Even one error was enough to fail VS Code test discovery. So the fix is either disable the linter or fix linter errors. I use setup described here.

Answer 18

This error is so frustrating.. In my case the error fixed by modifying the python.pythonPath parameter in settings.json (found inside .vscode folder under project root directory), to the path which I obtained using which python in terminal (e.g. /usr/local/var/pyenv/shims/python)

I use pyenv with python3.9, and my error said previously:

Error: Process returned an error: /Users/"user-name"/.pyenv/shims/python: line 21: /usr/local/Cellar/pyenv/1.2.23/libexec/pyenv: No such file or directory

at ChildProcess. (/Users/"user-name"/.vscode/extensions/littlefoxteam.vscode-python-test-adapter-0.6.8/out/src/processRunner.js:35:36) at Object.onceWrapper (events.js:422:26) at ChildProcess.emit (events.js:315:20) at maybeClose (internal/child_process.js:1021:16) at Process.ChildProcess._handle.onexit (internal/child_process.js:286:5)

Answer 19

I had this problem and struggle with it for hours. I think that there is a special resolution for every other platform configuration. My platform is:

• VSCode: 1.55.0 (Ubuntu)
• Pytest: 6.2.3
• MS Python extension (ms-python.python 2021.3.680753044)
• Python Test Explorer for Visual Studio Code (littlefoxteam.vscode-python-test-adapter - 0.6.7)

The worst thing has that the tool itself does not have a standard output (at least that I know of or could find easily on the internet).

In the end, the problem was

--no-cov

parameters that was not recognized by the VSCode testing explorer tool (that I copied from some page on the internet) and the error was showed by the extension littlefoxteam.vscode-python-test-adapter and it may help you to find where things are broken.

Answer 20

I searched in the settings for "python" and found this:

Switching to pytest automatically detect my tests:

Answer 21

If you are having trouble with pytest I was struggling with the discovery test part.

Reading some open issue (other) in vscode I found a workaround using the test-adapter extension

market_place_link

The extention works like a charm. (and solve my discovery problem)

Answer 22

I just thought I would add my answer here as this might well affect someone who uses a .env file for their project's environment settings since it is such a common configuration for 12 factor apps.

My example assumes that you're using pipenv for your virtual environment management and that you have a .env file at the project's root directory.

My vscode workspace settings json file looks like below. The crucial line for me here was "python.envFile": "${workspaceFolder}/.env",

{
    "python.pythonPath": ".venv/bin/python",
    "python.linting.enabled": true,
    "python.linting.pylintEnabled": true,
    "python.linting.pycodestyleEnabled": false,
    "python.linting.flake8Enabled": false,
    "python.linting.pylintPath": ".venv/bin/pylint",
    "python.linting.pylintArgs": [
        "--load-plugins=pylint_django",
    ],
    "python.formatting.provider": "black",
    "python.formatting.blackArgs": [
        "--line-length",
        "100"
    ],
    "python.testing.unittestEnabled": false,
    "python.testing.nosetestsEnabled": false,
    "python.testing.pytestEnabled": true,
    "python.testing.pytestPath": ".venv/bin/pytest",
    "python.envFile": "${workspaceFolder}/.env",
    "python.testing.pytestArgs": [
        "--no-cov"
    ],
}

I hope this saves someone the time I spent figuring this out.

Answer 23

Before begin the tests discovery, check that python.testing.cwd points correctly to your tests dir and your python.testing.pytestEnabled is set to true.

Once those requirements are set correctly, run tests discovery and its output (see OUTPUT window). You should see something like this:

python $HOME/.vscode/extensions/ms-python.python-$VERSION/pythonFiles/testing_tools/run_adapter.py discover pytest -- --rootdir $ROOT_DIR_OF_YOUR_PROJECT -s --cache-clear
Test Discovery failed: 
Error: ============================= test session starts ==============================
<SETTINGS RELATED TO YOUR MACHINE, PYTHON VERSION, PYTEST VERSION,...>
collected N items / M errors
...

It's important to highlight here the last line: collected N items / M errors. The following lines will contain info about the tests discovered by pytest. So your tests are discoverable but there are errors related to their correct execution.

The following lines will contain the errors which in most of the cases will be related to an incorrect import.

Check that all your dependencies have been downloaded previously. If you are working with specific versions of dependencies (on therequirements.txt file you have something like your_package == X.Y.Z) be sure that it's the right version that you need to.

Answer 24

I had this same issue and tracked it down to my pytest.ini file. Removing most of the addopts from that file fixed the issue.

Answer 25

Seems like a bug in the latest version of VS Code Python extension. I had the same issue, then I downgraded the Python extension to 2019.3.6558 and then it works again. So we should go to our VS Code extensions list, select the Python extension and "Install another version..." from the setting of that extension.

I hope this works for you too.

Answer 26

I resolved the issue by upgrading pytest to the latest version: 4.4.1 with "pip install --upgrade pytest". I was apparently running an old version 3.4.2

Answer 27

This is not a complete answer as I do not know why this is happening and may not relate to your problem, depending how you have your tests structured.

I resolved this issue by putting an __init__.py file in my tests folder

E.G.:

├───.vscode
│       settings.json
│
├───app
│       myapp.py
│
└───tests
        test_myapp.py
        __init__.py

this was working a few days ago without this but the python extension was recently updated. I am not sure if this is the intended behavior or a side effect of how discoveries are now being made

https://github.com/Microsoft/vscode-python/blob/master/CHANGELOG.md
Use Python code for discovery of tests when using pytest. (#4795)