pytest cannot import module while python can

Question

I am working on a package in Python. I use virtualenv. I set the path to the root of the module in a .pth path in my virtualenv, so that I can import modules of the package while developing the code and do testing (Question 1: is it a good way to do?). This works fine (here is an example, this is the behavior I want):

(VEnvTestRc) zz@zz:~/Desktop/GitFolders/rc$ python
Python 2.7.12 (default, Jul  1 2016, 15:12:24) 
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from rc import ns
>>> exit()
(VEnvTestRc) zz@zz:~/Desktop/GitFolders/rc$ python tests/test_ns.py 
issued command: echo hello
command output: hello

However, if I try to use PyTest, I get some import error messages:

(VEnvTestRc) zz@zz:~/Desktop/GitFolders/rc$ pytest
=========================================== test session starts ============================================
platform linux2 -- Python 2.7.12, pytest-3.0.5, py-1.4.31, pluggy-0.4.0
rootdir: /home/zz/Desktop/GitFolders/rc, inifile: 
collected 0 items / 1 errors 

================================================== ERRORS ==================================================
________________________________ ERROR collecting tests/test_ns.py ________________________________
ImportError while importing test module '/home/zz/Desktop/GitFolders/rc/tests/test_ns.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
tests/test_ns.py:2: in <module>
    from rc import ns
E   ImportError: cannot import name ns
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 errors during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
========================================= 1 error in 0.09 seconds ==========================================
(VEnvTestRc) zz@zz:~/Desktop/GitFolders/rc$ which pytest
/home/zz/Desktop/VirtualEnvs/VEnvTestRc/bin/pytest

I am a bit puzzled, it looks like this indicates an import error, but Python does it fine so why is there a problem specifically with PyTest? Any suggestion to the reason / remedy (Question 2)? I googled and stack-overflowed the 'ImportError: cannot import' error for PyTest, but the hits I got were related to missing python path and remedy to this, which does not seem to be the problem here. Any suggestions?

Answer 1

After correcting the import at the begging of the script (by appending my PYTHONPATH), so my module is found by pytest, I faced the issue of: Some test functions worked whereas some other were raising the "ModuleNotFoundError".

The issue was in the @patch("path.to.import.mymodule"): this path was incomplete. It needs the same syntax as the imports in the beggining of the script.

Answer 2

2024-02-10 PATH AND VISIBILITY MYSTERIES EXPLAINED

I originally answered this in October 2021. I had struggled with this problem for ages. As described in my original answer below NONE of the solutions here worked. Over 2 years I got 24 upvotes (and 2 downvotes) so it seemed like others were having similar intractable difficulties. I had reluctantly concluded that I had to manipulate sys.path to get pytest to be able to "see" the files that I thought it should see.

I was wrong and failed to understand something about paths. And CRUCIALLY this involves the running of a project (as most of my projects are) where the application is run by using this command (in W10):

> python src\core

In other words, from the start, the CWD is not the current (project root) directory, but a subdirectory [project root directory]\src\core (which contains file __main__.py)

This is where the problem arose. If you are not in fact running a file in the root directory of your project you need to be aware that putting the tests directory in your root directory will plunge you into a world of pain and bafflement: sibling .py files will bafflingly not be "found" in your main "run" directory, i.e. where CWD is, i.e. by virtue of your command line.

Instead, in my case, I need the root "tests" directory to be a subdirectory of src\core (relative to the root directory of my project).

As a more general rule: wherever the CWD of your project is as you run it as an application is the place where you must put your "tests" directory.

Hope this helps someone.

-------------- MY ORIGINAL NAIVE, INCORRECT ANSWER OF 2021-10:

ANOTHER SUGGESTION

I explored this question and various others on SO and elsewhere... all the stuff about adding (or removing) an empty __init__.py in and/or conftest.py in various parts of the project directory structure, all the stuff about PYTHONPATH, etc., etc.: NONE of these solutions worked for me, in what is actually a very simple situation, and which shouldn't be causing any grief.

I regard this as a flaw in pytest's current setup. In fact I got a message recently from someone on SO who clearly knew his stuff. He said that pytest is not designed to work with (as per Java/Groovy/Gradle) separate "src" and "test" directory structures, and that test files should be mingled in amongst the application directories and files. This perhaps goes some way to providing an explanation ... however, tests, particularly integration/functional tests, don't always necessarily correspond neatly to particular directories, and I think pytest should give users more choice in this regard.

Structure of my project:

project_root
    src
        core
            __init__.py
            __main__.py
            my_other_file.py 
    tests
        basic_tests
            test_first_tests.py

The import problem posed: very simply, __main__.py has a line import my_other_file. This (not surprisingly) works OK when I just run the app, i.e. run python src/core from the root directory.

But when I run pytest with a test which imports __main__.py I get

ModuleNotFoundError: No module named 'my_other_file'

on the line in __main__.py that tries to import "my_other_file". Note that the problem here, in my case, is that, in the course of a pytest test, one application file fails to find another application file in the same package.

USING PYTHONPATH
After a lot of experimentation, and putting an __init__.py file and a confest.py file in just about every directory I could find (I think the crucial files were __init__.py added to "tests" and "basic_tests", see above directory structure), and then setting PYTHONPATH to as follows

PYTHONPATH=D:\My Documents\software projects\EclipseWorkspace\my_project\src\core

... I found it worked. Imports in the testing files had to be tweaked a bit, the general pattern being from core import app, project, but the test files were able to "see" core, and crucially there was no need to mess around with the import commands in the app files.

HOWEVER... for some reason the tests now run much slower using this method! Compared to my solution below, where the core package can be seen to be loaded just once, my suspicion is that the PYTHONPATH solution probably results in vast amounts of code being reloaded again and again. I can't yet confirm this, but I can't see any other explanation.

THE ALTERNATIVE

As I say, I regard this as a flaw in pytest's setup. I have absolutely no idea what pytest does to establish a common-sense setup for sys.path, but it's obviously getting it wrong. There should be no need to rely on PYTHONPATH, or whatever, and if indeed this is the "official" solution, the documentation on this subject is sorely lacking.

The ONLY working solution, at least on my machines (OS W10 and Linux, as at pytest 7.3.1), is to add the application packages to sys.path. It is undesirable to mess with sys.path, but in this case nothing else at all works for me.

The most obvious place to do this is conftest.py, which gets executed at the start of each pytest run, and which is kept the project root directory. It involves no modification of the app files. Typically (for the above directory/files setup):

this_file_path = pathlib.Path(__file__)
src_core_dir_path_str = str(this_file_path.parent.joinpath('src', 'core'))
sys.path.insert(0, src_core_dir_path_str)

I am currently (2023-08) using pytest v 7.3.1, and have also tried editing pytest.ini, as suggested by ian's 2022 answer, where it is suggested that these configurations in pytest.ini are a new thing to solve the problem since v7 (release 2021-12). In my machines this does not solve the problem: without manually altering sys.path pytest's attempt to import __main__.py in my example fails as before: because import my_other_file raises ModuleNotFoundError: No module named 'my_other_file'.

PS I have now set up a minimal git repo illustrating the problem: download the .zip at https://github.com/Mrodent/pytest_probs/tree/main, and read what I say in README.md. Interested to know if anyone gets different results to what I get.

--

Answer 3

As of pytest 7.0, you can now add pythonpath in pytest.ini. No need to add __init__.py or conftest.py in your root directory.

[pytest]
minversion = 7.0
addopts = --cov=src
pythonpath = src
testpaths =
    tests

You can run pytest without any parameters.

https://docs.pytest.org/en/7.0.x/reference/reference.html#confval-pythonpath

Update:

Here's a sample repo on how I setup my projects which uses the above config:

https://github.com/ianpogi5/pytest-path

Answer 4

I can generally recommend using absolute paths and running the applications and/or tests from the root or perhaps parent folder.

If someone is receiving this error in FastAPI with Uvicorn, simply run e.g.

uvicorn app.main:app --host 0.0.0.0 --port 8080

Answer 5

The TestCase directory must be a Python Package

Answer 6

I faced the same/similar issue when trying to import a new class in my test file. In my case I moved the new import to be the last one in the test file, and then the problem went away. Not entirely sure why though.

Answer 7

Points to be observed in the same order as below:

1. Make sure that every folder and subfolder of your app has a __init__.py file.
2. Make sure that the tests folder(module) is inside the app folder(module).
3. Try pytest command and if that doesn't work try python3 -m pytest .

Answer 8

My issue was solved by fixing an import. I was importing from project_root.tests.xxxx and when I changed it to from tests.xxxx it worked. Funny thing is Pycharm didn't mark it as an error in the project tree view, I had to go into the file and only there it showed in red.

Answer 9

my case:

• all configurations are correct except I forgot to remove the default my_app/tests.py

  • After removing it, python -m pytest tests/my_app works fine

• pytest.ini:

[pytest]
DJANGO_SETTINGS_MODULE=necktie_hy.settings
python_files=*/tests/*/test_*.py
python_classes=Test

Answer 10

I was having the same problem using pytest under tox. I solved it by including any submodule of my project in my setup.cfg file.

My project structure:

src/
    main_module
       - __init__.py
       ...
       submodule_a
           - __init__.py
           ...
       submodule_b
           - __init__.py
           ...
tests/
    - __init__.py
    - test_a.py
    ...

My setup.cfg file

...
[options]
packages = main_module, main_module.submodule_a, main_module.submodule_b
package_dir = 
    = src
...

Answer 11

For me, this issue was that my test file was named: xxx.test.py

it was resolved my naming the file: xxx_test.py

But i imagine naming it: test_xxx.py would work too.

Answer 12

I solved my problem by setting the PYTHONPATH in Environment Variables for the specific configuration I'm running my tests with.

While you're viewing the test file on PyCharm:

1. Ctrl + Shift + A
2. Type Edit Configurations
3. Set your PYTHONPATH under Environment > Environment variables.

UPDATE

1. Move into your project's directory root
2. Create a virtual environment
3. Activate your newly created virtual environment
4. Set the variable $PYTHONPATH to the root of your project and export it:

export PYTHONPATH=$(pwd)

5. Do not remove the __init__.py from the tests/ directory or from the src/ directory.

Also note:

• The root of your directory is not a python module so do NOT add an __init__.py
• conftest.py is not necessary in the root of your project.

6. The $PYTHONPATH var will only be available during the current terminal/console session; so you will need to set this each time. (You can follow the steps previous to this update if you are working in pycharm).

Answer 13

update PYTHONPATH till src folder

export PYTHONPATH=/tmp/pycharm_project_968/src

Answer 14

I just solved this by removing the __init__.py in my project root:

.
├── __init__.py <--- removed
├── models
│   ├── __init__.py
│   ├── address.py
│   ├── appointment.py
│   └── client.py
├── requirements.txt
├── setup.cfg
├── tests
│   ├── __init__.py
│   ├── models
│   │   ├── __init__.py
│   │   ├── appointment_test.py
│   │   └── client_test.py
│   └── other_test.py
└── script.py

Answer 15

Kept everything same and just added a blank test file at the root folder .. Solved it

Here are the findings, this problem really bugged me for a while. My folder structure was

mathapp/
    - server.py  
    - configuration.py 
    - __init__.py 
    - static/ 
       - home.html  
tests/            
    - functional 
       - test_errors.py 
    - unit  
       - test_add.py

and pytest would complain with the ModuleNotFoundError and gives the hint:

make sure your test modules/packages have valid Python names.

I introduced a mock test file at the same level as mathsapp and tests directory. The file contained nothing. Now pytest does not complain.

Result without the file

$ pytest
============================= test session starts =============================
platform win32 -- Python 3.8.2, pytest-5.4.2, py-1.8.1, pluggy-0.13.1
rootdir: C:\mak2006\workspace\0github\python-rest-app-cont
collected 1 item / 1 error

=================================== ERRORS ====================================
_______________ ERROR collecting tests/functional/test_func.py ________________
ImportError while importing test module 'C:\mainak\workspace\0github\python-rest-app-cont\tests\functional\test_func.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
tests\functional\test_func.py:4: in <module>
    from mathapp.service import sum
E   ModuleNotFoundError: No module named 'mathapp'
=========================== short test summary info ===========================
ERROR tests/functional/test_func.py
!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
============================== 1 error in 0.24s ===============================

Results with the file

$ pytest
============================= test session starts =============================
platform win32 -- Python 3.8.2, pytest-5.4.2, py-1.8.1, pluggy-0.13.1
rootdir: C:\mak2006\workspace\0github\python-rest-app-cont
collected 2 items

tests\functional\test_func.py .                                          [ 50%]
tests\unit\test_unit.py .                                                [100%]

============================== 2 passed in 0.11s ==============================

Answer 16

Simply put an empty conftest.py file in the project root directory, because when pytest discovers a conftest.py, it modifies sys.path so it can import stuff from the conftest module. A general directory structure can be:

Root
├── conftest.py
├── module1
│   ├── __init__.py
│   └── sample.py
└── tests
    └── test_sample.py

Answer 17

I had placed all my tests in a tests folder and was getting the same error. I solved this by adding an __init__.py in that folder like so:

.
|-- Pipfile
|-- Pipfile.lock
|-- README.md
|-- api
|-- app.py
|-- config.py
|-- migrations
|-- pull_request_template.md
|-- settings.py
`-- tests
    |-- __init__.py <------
    |-- conftest.py
    `-- test_sample.py

Answer 18

I find the answer in there :Click here

If you have other project structure, place the conftest.py in the package root dir (the one that contains packages but is not a package itself, so does not contain an init.py)

Answer 19

This can be resolved by adding the respective path to the System's Environmental Variables.

Since you are using the virtualenv, the pytest may not read the environmental variables of the virtual machine.

I also faced this issue and resolved it by adding the path to the System's Environmental Variable (PATH)

Answer 20

Please check here: https://docs.pytest.org/en/documentation-restructure/background/pythonpath.html

I has an issue with pytest (that was solved using python -m pytest); the error was

FileNotFoundError: [Errno 2] No such file or directory: '/usr/local/lib/python3.9/site-packages/...

I found the problem was missing __init__.py in tests/ and tests/subfolders.

Answer 21

Here's a medium article! describing the problem!

The issue is which pytest you are using and your use of a virtual environment. If you have installed pytest system-wide, in other words, outside of a virtual environment, pytest has a nasty habit of only looking outside your virtual environment for modules! If your project is using a module that is only installed in your virtual environment, and you’re using a system-wide pytest, it won’t find the module, even if you’ve activated the virtual environment.1

Here’s the step-by-step:1

1. Exit any virtual environment
2. Use Pip to uninstall pytest
3. Activate your project’s virtual environment
4. Install pytest inside the virtual environment
5. pytest will now find your virtual-environment-only packages!

Answer 22

In my case the problem was that the filename and the class name were exactly the same: FrameAnalyzer.py and FrameAnalyzer respectively. Once I changed the filename to frame_analyzer.py, everything worked.

Answer 23

I had a similar problem just recently. The way it worked for me it was realizing that "setup.py" was wrong

Previously I deleted my previous src folder, and added a new one with other name, but I didn't change anything on the setup.py (newbie mistake I guess).

So pointing setup.py to the right packages folder did the trick for me

from setuptools import find_packages, setup

setup(
    name="-> YOUR SERVICE NAME <-",
    extras_Require=dict(test=["pytest"]),
    packages=find_packages(where="->CORRECT FOLDER<-"),
    package_dir={"": "->CORRECT FOLDER<-"},
)

Also, not init.py in test folder nor in the root one.

Hope it helps someone =)

Best!

Answer 24

If you run Pytest from a terminal:

Run pytest with the --import-mode=append command-line flag.

Argument description in the official documentation: https://docs.pytest.org/en/stable/pythonpath.html

---

UPD: Before I also wrote how to do the same if you use PyCharm, but community does not like extendend answers, so I removed additional information that probably was helpful to someone who have a similar issue.

Answer 25

In my case I am working in a container and unfortuantely pytest has the tendency to use python2.7 rather than my python3 interpreter of choice.

In my case this worked:

python3 -m pytest

My folder structure

/
app/
-module1.py
-module2.py
-tests/
--test_module1.py
--test_module2.py
requirements.txt
README.md

Answer 26

Had a similar issue and it worked when I added __init__.py file under tests directory.

Answer 27

I disagree with the posts saying that you must remove any __init__.py files. What you must instead do is alter the sys.path.

Run an experiment where you print sys.path when running the code normally. Then print sys.path while running the code via pytest. I think you will find that there is a difference between these two paths, hence why pytest breaks.

To fix this, insert the path from the first experiment at the 0th index of the second.

Let '/usr/exampleUser/Documents/foo' be the first element of print(sys.path) for experiment 1.

Below is code that should fix your issue:

import sys sys.path[0] = '/usr/exampleUser/Documents/foo'

Put this at the top of your file, before your actual import statement.

Source: I was dealing with this myself and the above process solved it.

Answer 28

My 2 cents on this: pytest will fail at chance if you are not using virtual environments. Sometimes it will just work, sometimes not.

Therefore, the solution is:

• remove pytest with pip uninstall
• create your venv
• activate your venv
• pip install your project path in editable mode, so it will be treated by pytest as a module (otherwise, pytest wont find your internal imports). You will need a setup.py file for that
• install your packages, including pytest
• finally, run your tests

The code, using windows PowerShell:

pip uninstall pytest
python.exe -m venv my_env
.\my_env\Scripts\activate
(my_env) pip install -e .
(my_env) pip install pytest pytest-html pandas numpy

Then finally

(my_env) pytest --html="my_testing_report.html"

An example of setup.py, for pip install -e:

import setuptools

setuptools.setup(
    name='my_package',
    version='devel',
    author='erickfis',
    author_email='erickfis@gmail.com',
    description='My package',
    long_description='My gooood package',
    packages=setuptools.find_packages(),
    classifiers=[
        'Programming Language :: Python :: 3',
        'Operating System :: OS Independent',
    ],
    include_package_data=True
)

Answer 29

Yet another massive win for Python's import system. I think the reason there is no consensus is that what works probably depends on your environment and the tools you are using on top of it.

I'm using this from VS Code, in the test explorer under Windows in a conda environment, Python 3.8.

The setup I have got to work is:

mypkg/
    __init__.py
    app.py
    view.py
tests/
    test_app.py
    test_view.py

Under this setup intellisense works and so does test discovery.

Note that I originally tried the following, as recommended here.

src/
    mypkg/
        __init__.py
        app.py
        view.py
tests/
    test_app.py
    test_view.py

I could find no way of getting this to work from VS Code because the src folder just blew the mind of the import system. I can imagine there is a way of getting this to work from the command line. As a relatively new convert to Python programming it gives me a nostalgic feeling of working with COM, but being slightly less fun.

Answer 30

I was experiencing this issue today and solved it by calling python -m pytest from the root of my project directory.

Calling pytest from the same location still caused issues.

My Project dir is organized as:

api/
 - server/
  - tests/
      - test_routes.py
  - routes/
      - routes.py
 - app.py

The module routes was imported in my test_routes.py as: from server.routes.routes import Routes

Hope that helps!