Import problem when separating applications and package tests

Question

Maybe my goal and what I try to do here is wrong in the meaning of unpythonic. I am open for any suggestions about that.

My goals

1. Application (myapp) with its own tests folder.
2. A package (mypackage) with its own tests folder.
3. The tests for the package should be runable from the application folder and from the package folder.
4. The package have implicit and explicit components. The latter need to be imported explicit (e.g. via import mypackage.mymoduleB).
5. The package (folder) can be copied (shipped for reuse in other applications?) to other file system locations without loosing its functionality and testabilibty. That is why tests is inside the package folder and not outside.

That is the folder tree where itest is the name of the project, myapp is the application with an if __name__ == '__main__': in it and mypackag is the package.

itest
└── myapp
    ├── myapp.py
    ├── mypackage
    │   ├── __init__.py
    │   ├── _mymoduleA.py
    │   ├── mymoduleB.py
    │   └── tests
    │       ├── __init__.py
    │       └── test_all.py
    └── tests
        ├── __init__.py
        └── test_myapp.py

The problem

I can run the unittests from the application directory without problems.

/home/user/tab-cloud/_transfer/itest/myapp $ python3 -m unittest -vvv
test_A (mypackage.tests.test_all.TestAll) ... mymoduleA.foo()
ok
test_B (mypackage.tests.test_all.TestAll) ... mymoduleB.bar()
ok
test_myname (tests.test_myapp.TestMyApp) ... ok

----------------------------------------------------------------------
Ran 3 tests in 0.001s

OK

But when I go inside the package the tests do not run (sieh goal #3).

/home/user/tab-cloud/_transfer/itest/myapp/mypackage $ python3 -m unittest -vvv
tests.test_all (unittest.loader._FailedTest) ... ERROR

======================================================================
ERROR: tests.test_all (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: tests.test_all
Traceback (most recent call last):
  File "/usr/lib/python3.9/unittest/loader.py", line 436, in _find_test_path
    module = self._get_module_from_name(name)
  File "/usr/lib/python3.9/unittest/loader.py", line 377, in _get_module_from_name
    __import__(name)
  File "/home/user/tab-cloud/_transfer/itest/myapp/mypackage/tests/test_all.py", line 12, in <module>
    from . import mypackage
ImportError: cannot import name 'mypackage' from 'tests' (/home/user/tab-cloud/_transfer/itest/myapp/mypackage/tests/__init__.py)


----------------------------------------------------------------------
Ran 1 test in 0.001s

FAILED (errors=1)

The MWE

No I show you the files. To make sure the tests for the package using the right import when run from the application folder or from the package folder I use importlib (based on foreign solution).

The three files form the package

This is myapp/mypackage/__init__.py:

# imported implicite via 'mypackage'
from ._mymoduleA import *

# 'mymoduleB' need to be imported explicite
# via 'mypackage.moduleB'

This is myapp/mypackage/_mymoduleA.py:

def foo():
    print('mymoduleA.foo()')
    return 1

This is myapp/mypackage/mymoduleB.py:

def bar():
    print('mymoduleB.bar()')
    return 2

The tests for the package

The myapp/mypackage/tests/__init__.py is empty.

This is myapp/mypackage/tests/test_all.py:

import importlib
import unittest

# The package should be able to be tested by itself (run unittest inside the
# package directory) AND from the using application (run unittest in
# application directory).
# Based on: https://stackoverflow.com/a/14050282/4865723
if importlib.util.find_spec('mypackage'):
    import mypackage
    import mypackage.mymoduleB
else:
    from . import mypackage
    from mypackage import mymoduleB


class TestAll(unittest.TestCase):
    def test_A(self):
        self.assertEqual(1, mypackage.foo())

    def test_B(self):
        self.assertEqual(2, mypackage.mymoduleB.bar())

The application

This is cat myapp/myapp.py:

#!/usr/bin/env python3

import mypackage


def myname():
    return 'My application!'


if __name__ == '__main__':
    print(myname())

    mypackage.foo()

    try:
        mypackage.mymoduleB.bar()
    except AttributeError:
        # we expecting this
        print('Not imported yet: "mymoduleB.bar()"')

    # this should work
    import mypackage.mymoduleB
    mypackage.mymoduleB.bar()

The test for the application

The myapp/tests/__init__.py is empty.

This is myapp/tests/test_myapp.py:

import unittest
import myapp


class TestMyApp(unittest.TestCase):
    def test_myname(self):
        self.assertEqual(myapp.myname(), 'My application!')

Sidenotes

Please let me explain something more about my goals. The mypackage should be reusable in other projects. In practice this means I copy the mypackage folder from one place to another. And while copy that folder I do want that tests folder come with it without explicte thinking about it because it is outside the package folder. And if the new project does unittesting the tests of the package should be involved in that unittesting automaticlly (via discover).

Answer 1

Your goal is really a bit unpythonic. But sometimes, you have to break the rules to free your heart.

You can solve the problem by checking for the __package__ attribute in myapp/mypackage/__init__.py like this:

# hint from there: https://stackoverflow.com/a/65426846/4865723
import sys
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parent.parent))

if __package__:
    from ._mymoduleA import foo
else:
    from _mymoduleA import *
    

In this case myapp/mypackage/tests/test_all.py the code gets a little simpler:

import importlib
import unittest

if not importlib.util.find_spec('mypackage'):
    from __init__ import *

import mypackage
from mypackage import mymoduleB


class TestAll(unittest.TestCase):
    def test_A(self):
        self.assertEqual(1, mypackage.foo())

    def test_B(self):
        self.assertEqual(2, mymoduleB.bar())

All other files remain unchanged.

As a result, you get the ability to run tests from both /myapp and /myapp/mypackage folder. At the same time, there is no need to hardcode any absolute paths. The app can be copied to any other file system locations.

I hope it will useful for you.

Answer 2

I created an import library a couple of years ago. It works on pathing. I used it to create a plugin system where I could essentially install and import multiple versions of any library (with some limitations).

For this we get the current path of the module. Then we import the package using the path. This library will automatically add the proper path to sys.path.

All you need to do is install pylibimp pip install pylibimp and edit myapp/mypackage/tests/test_all.py

import os
import pylibimp
import unittest


path_tests = os.path.join(os.path.dirname(__file__))
path_mypackage = os.path.dirname(path_tests)
path_myapp = os.path.dirname(path_mypackage)
mypackage = pylibimp.import_module(os.path.join(path_myapp, 'mypackage'), reset_modules=False)


class TestAll(unittest.TestCase):
    def test_A(self):
        self.assertEqual(1, mypackage.foo())

    def test_B(self):
        self.assertEqual(2, mypackage.mymoduleB.bar())

I believe the background is fairly simple.

import os
import sys

sys.path.insert(0, os.path.abspath('path/to/myapp'))

# Since path is added we can "import mypackage"
mypackage = __import__('mypackage')

sys.path.pop(0)  # remove the added path to not mess with other imports

I hope this is what you are looking for.