How to run unitests of the form test/a.py?

Question

Is it possible to implement a Python project with a file structure like the following?:

myproj
├── a.py
├── b.py
├── c.py
└── test/
    ├── a.py
    ├── b.py
    └── c.py

Note, in particular, that the test scripts under test/ have the same basenames as the module files they are testing  ¹. (In other words, test/a.py contains the unit tests for a.py; test/b.py contains those for b.py, etc.)

The tests under test/ all import unittest and define subclasses of unittest.TestCase.

I want to know how to run the tests under test/, both individually, and all together.

I've tried many variations of python -m unittest ..., but they all either fail (examples below), or end up running zero tests.

For example,

% python -m unittest test.a
Traceback (most recent call last):
  File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/usr/lib/python2.7/unittest/__main__.py", line 12, in <module>
    main(module=None)
  File "/usr/lib/python2.7/unittest/main.py", line 94, in __init__
    self.parseArgs(argv)
  File "/usr/lib/python2.7/unittest/main.py", line 149, in parseArgs
    self.createTests()
  File "/usr/lib/python2.7/unittest/main.py", line 158, in createTests
    self.module)
  File "/usr/lib/python2.7/unittest/loader.py", line 130, in loadTestsFromNames
    suites = [self.loadTestsFromName(name, module) for name in names]
  File "/usr/lib/python2.7/unittest/loader.py", line 100, in loadTestsFromName
    parent, obj = obj, getattr(obj, part)
AttributeError: 'module' object has no attribute 'a'

If I change the name of the test/ directory to t/, then the error becomes:

% python -m unittest t.a
Traceback (most recent call last):
  File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/usr/lib/python2.7/unittest/__main__.py", line 12, in <module>
    main(module=None)
  File "/usr/lib/python2.7/unittest/main.py", line 94, in __init__
    self.parseArgs(argv)
  File "/usr/lib/python2.7/unittest/main.py", line 149, in parseArgs
    self.createTests()
  File "/usr/lib/python2.7/unittest/main.py", line 158, in createTests
    self.module)
  File "/usr/lib/python2.7/unittest/loader.py", line 130, in loadTestsFromNames
    suites = [self.loadTestsFromName(name, module) for name in names]
  File "/usr/lib/python2.7/unittest/loader.py", line 91, in loadTestsFromName
    module = __import__('.'.join(parts_copy))
ImportError: No module named t

Or

% python -m unittest t/a.py
Traceback (most recent call last):
  File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/usr/lib/python2.7/unittest/__main__.py", line 12, in <module>
    main(module=None)
  File "/usr/lib/python2.7/unittest/main.py", line 94, in __init__
    self.parseArgs(argv)
  File "/usr/lib/python2.7/unittest/main.py", line 149, in parseArgs
    self.createTests()
  File "/usr/lib/python2.7/unittest/main.py", line 158, in createTests
    self.module)
  File "/usr/lib/python2.7/unittest/loader.py", line 130, in loadTestsFromNames
    suites = [self.loadTestsFromName(name, module) for name in names]
  File "/usr/lib/python2.7/unittest/loader.py", line 91, in loadTestsFromName
    module = __import__('.'.join(parts_copy))
ImportError: Import by filename is not supported.

(I am using Python 2.7.9.)

---

UPDATE

Since I put a bounty on this question, I'll be very explicit about what would constitute an acceptable answer.

Either one of the following would be acceptable:

• How to invoke unittest from the command line to run either individual tests, or all the tests under the test/ directory; it is acceptable for solution to entail small changes to the code in the test scripts (e.g. modifications to the import statements).

• If the file structure shown above is not possible for some reason, a detailed explanation would be an acceptable solution.

As a base case, start with the following minimal case with the following file structure:

myproj
├── a.py
├── b.py
└── test/
    ├── a.py
    └── b.py

...and the following contents

# a.py
def hello():
    print 'hello world'

# b.py
def bye():
    print 'good-bye world'

# test/a.py

import unittest
import a

class TestA(unittest.TestCase):
    def test_hello(self):
        self.assertEqual(a.hello(), None)

# test/b.py

import unittest
import b

class TestB(unittest.TestCase):
    def test_bye(self):
        self.assertEqual(b.bye(), None)

Show how one tells unittest to run the test test/a.py, and how to run "all the tests under test". (The latter should continue to work, even if the new test scripts are added to test, or some of the current test scripts are removed.)

---

Minimal tests of the proposals offered so far show that they don't work. For example:

% python -m unittest discover -s test -p '*.py'
EE
======================================================================
ERROR: test_hello (a.TestA)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/SHIVAMJINDAL/myproj/test/a.py", line 6, in test_hello
    self.assertEqual(a.hello(), None)
AttributeError: 'module' object has no attribute 'hello'

======================================================================
ERROR: test_bye (b.TestB)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/SHIVAMJINDAL/myproj/test/b.py", line 6, in test_bye
    self.assertEqual(b.bye, None)
AttributeError: 'module' object has no attribute 'bye'

----------------------------------------------------------------------
Ran 2 tests in 0.000s

FAILED (errors=2)
% tree .
.
├── a.py
├── b.py
├── __init__.py
└── test/
    ├── a.py
    ├── b.py
    └── __init__.py

1 directory, 6 files

---

^{1 This constraint is very much intentional, and it is an integral part of the problem presented here. (IOW, a "solution" that entails relaxing this constraint is in fact not a solution.)}

Answer 1

I had similar requirements some time ago (python 2.6) and I ended up implementing a little executable script that will run your tests (call it run_tests, code below) by loading modules dynamically based on file paths.

The only change that you will have to make is to put your source modules into a package, which makes a lot of sense since you want to import these source modules from your test modules.

If you really want to have your source modules in the root of your project, it is possible with this solution as well, but you will then have to rename your test modules to something different (like test_a.py for instance). I can tweak the current solution to make this work if you want me to.

Your project structure needs to look like this:

.
├── project
│   ├── a.py
│   └── __init__.py
└── test
    ├── a.py
    └── run_tests

Here, project is the package containing your source a.py, remember, to make it a package, the project folder needs an empty __init__.py file in it.

Now the content of run_tests:

#!/usr/bin/env python

import fnmatch
import imp
import os
import sys
import unittest

append_lib = lambda p: sys.path.append(p)
root_dir = os.path.join(os.path.dirname(os.path.realpath(__file__)), "..")
test_dir = os.path.join(root_dir, "test")

append_lib(root_dir) # packages to be tested are to be found there

def find_files():
  matches = []
  for root, dirnames, filenames in os.walk(test_dir):
    for filename in fnmatch.filter(filenames, "*.py"):
      matches.append(os.path.join(root, filename))
  return matches

def module_from_path(path):
  module = os.path.splitext(os.path.basename(path))[0]
  return imp.load_source(module, path)

def run_tests_in_paths(paths):
  suite = unittest.TestSuite()
  for p in paths:
    s = unittest.findTestCases(module_from_path(p))
    suite.addTests(s)
  if not unittest.TextTestRunner(verbosity=2).run(suite).wasSuccessful():
    raise RuntimeError("some test cases failed")

if __name__ == "__main__":
  if len(sys.argv) > 1:
    paths = sys.argv[1:]
    print "> Running %s individual test file%s" % (len(paths), "s" if len(paths) > 1 else "")
    run_tests_in_paths(sys.argv[1:])
  else:
    print "> Running complete test suite"
    run_tests_in_paths(find_files())

I tested this setup with this content for project/a.py:

class A(object):
  pass

And for test/a.py:

import unittest

from project import a

class TestA(unittest.TestCase):
  def test_hello(self):
    self.assertNotEqual(a.A(), None)

You can now run the run_tests script (do not forget to chmod +x it) like this for a single file:

./test/run_tests test/a.py

Or like that to run any test modules in the test folder:

./test/run_tests

Note: do not hesitate to improve / customize the run_tests scripts to better fit your needs.

I hope this helps!

Answer 2

I think the structure you show in your question is like that you are using some IDE such as Pycharm before.

In Pycharm, you absolutely can have that project structure. As you can run unittest in a single file by just right-clicking it and clicking Run 'unittest in a.py'. And you can run unittest in whole directory by right-clicking on directory and click Run 'unittest in test'.

Further, you needn't to create a __ini__.py in test to make it become a module and importable. Also you can directly import your_lib in test file without worrying about import path problem.

All of these are done by the magic of IDE, so you cannot directly achieve this only using python -m unittest .... If you can, why does Pycharm use a helper named _jb_unittest_runner.py?

But you can create a simple bash script to achieve similar behave.

The keys are:

• Assign project root to a variable such as PROJECT_HOME.
• Assign test root to a variable such as TEST_HOME.
• Add PROJECT_HOME to PYTHONPATH so that you can directly import your project in test files.
• Change your working directory to ${TEST_HOME}
• Use python -m unittest ${PROJECT_HOME}/test/${file_name} to run a single file.
• Use python -m unittest discover -s ${PROJECT_HOME}/test -t ${PROJECT_HOME}/test to run a whole directory.
• Use python -m unittest ${file_name}.${test_case} to run a single test case.

Answer 3

Is it possible to implement a Python project with a file structure like the following?

I don't think this structure would be usable in the way you are trying. In this apporach, the root problem is the module names conflict in your tests.

For example, the statement import a in your test case 'test\a.py' resolve to the module test\a instead of referring to module under test (a.py)

Since you want to name your test modules exactly same as the module under test, one possible solution could by that you move all your modules under a package and the change import statement in your test. For example:

+---python2unittest
¦       a.py
¦       b.py
¦       __init__.py
+---test
¦   ¦   a.py
¦   ¦   b.py

Your test case, test/a.py will look like:

import unittest

from python2unittest import a


class TestA(unittest.TestCase):

    def test_hello(self):
        self.assertEqual(a.hello(), None)

How to run tests from command line:

python -m unittest discover -s  -t test -p a.py  // Specific Test
python -m unittest discover -s  -t test -p *.py  // All tests

Note:- If you have sub-directories for your test files, you need to add the __init__.py file in each sub-directory so that all test files can be discovered by unittest.

More information on test discovery can be found here:

Answer 4

I'm fairly certain your issue is that you run Python 2.7, where doing a plain import a is ambiguous wether a means the one relative to the current module or from the base path. Add the line from __future__ import absolute_import to the top of your files (it has to be the first non-comment line). I'd recommend you do it on all files, but it should be enough to do it on the test/*.py files.

Once done, you should be able to run python -m unittest discover -s test -p '*.py'

Answer 5

I was able to make your approach work, but there are few changes that are needed and mandatory.

1. The test should be run with a parent directory context
2. Both main folder and test folder should have a __init__.py
3. The imports in the tests should be a relative import and not a direct import

So below is my tree structure

root@5db7ad85dafd:/project# tree
.
 __init__.py
 a.py
 test
     __init__.py
     a.py

1 directory, 4 files

root@5db7ad85dafd:/project# python --version
Python 2.7.9

project/a.py

hello = 'tarun'

project/test/a.py

import unittest
from .. import a

class TestStringMethods(unittest.TestCase):
   def test_abc(self):
       assert a.hello == "tarun"

Notice the from .. import a which is import for this to work

Next we run the test being in the root folder of the project like below

root@5db7ad85dafd:/project# python -m unittest discover -t .. -s test -p "*.py"
.
----------------------------------------------------------------------
Ran 1 test in 0.000s

OK

-t here sets the top level import directory, so our relative imports can work -s tells which directory our tests are in -p tells what patter the test should be discovered in

When you want to run a individual test you will do something like below

python -m unittest discover -t .. -s test -p "a.py"

or

python -m unittest discover -t .. -s test -p "*.py" a

A picture is always worth more than words

Edit-1

Wanted to update my answer after seeing Peter's answer. The reason I didn't mention the import from a fixed named package was that it would mean that you need to know the name of the folder where the code is cloned and it is enforced to remain the same. But if you still want to go with that approach, then one approach is to move the actual into a sub-folder

So it would be repo/project/test/a.py and then in your tests you will use

from project import a

and then run it like below from the repo folder

root@5db7ad85dafd:/repo# python -m unittest discover -v -t project -s project.test -p "*.py"
test_abc (test.a.TestStringMethods) ... ok

----------------------------------------------------------------------
Ran 1 test in 0.000s

OK

or like below from the test folder

root@5db7ad85dafd:/repo/project# python -m unittest discover -v -t .. -s test -p "*.py"
test_abc (project.test.a.TestStringMethods) ... ok

----------------------------------------------------------------------
Ran 1 test in 0.001s

OK

In this case moving your project folder one level from root, will make sure that the project name is not dependent on the folder where your project is cloned

Answer 6

Tarun's answer is already pretty complete... In short:

1. You must make your modules importable (by creating __init__.py files)
2. You must tell the interpreter exactly where to look to prevent the naming clash between your modules.

The import doesn't have to be relative, though. Assuming that your project is going to be packaged up the import could (arguably, should) be precisely as you would expect users of your package to use - e.g. from myproj import a.

At this point I have also got python -m unittest discover -t .. -s test -p '*.py' to work. But this is the point where I get fed up with the extra hoops that the basic unittest package places on the user. I'd also recommend that once you have made the 2 changes, you also install nosetests (strictly speaking the python nose package) as it generally makes life easier for you to find and run the tests.

For example:

$ tree
.
├── a.py
├── a.pyc
├── b.py
├── b.pyc
├── __init__.py
├── __init__.pyc
└── test
    ├── a.py
    ├── a.pyc
    ├── b.py
    ├── b.pyc
    ├── __init__.py
    └── __init__.pyc

1 directory, 12 files
$ cat test/a.py
# test/a.py

import unittest
from myproj import a

class TestA(unittest.TestCase):
    def test_hello(self):
        self.assertEqual(a.hello(), None)
$ nosetests test/*.py
..
----------------------------------------------------------------------
Ran 2 tests in 0.001s

OK

Answer 7

It seems you do not have __init__.py file under your python package myproj and test. that's you are not able to import a module in test of a.

Please add blank __init__.py file in test and myproj directory both. If this does not solve the problem, please provide code of any test file.

You have used import b in your test/b.py file but it should be from myproj import b. Because if you are using just import b it will import the current file (in which you have written the test and this file does not have any bye() method). So either you can change the file name of your test files or use below code.

# test/b.py

import unittest
from myproj import b

class TestB(unittest.TestCase):
    def test_bye(self):
        self.assertEqual(b.bye(), None)

Answer 8

You'd better package your app the setuptools way, this would make things easier with a regular package files layout.

Anayway you should try the auto discover mode:

• put an test/__init__.py file. Even empty.
• put a class inheriting unittest.TestCase in test/anything.py with test_xxx method(s)
• cd root/of/project
• python -m unittest discover -s test -p *.py

See https://docs.python.org/2.7/library/unittest.html#test-discovery about the used options.