Using doctests from within unittests

Question

I typically write both unittests and doctests in my modules. I'd like to automatically run all of my doctests when running the test suite. I think this is possible, but I'm having a hard time with the syntax.

I have the test suite

import unittest
class ts(unittest.TestCase):
    def test_null(self): self.assertTrue(True)
if __name__ == '__main__': unittest.main()

I'd like to add to this suite all of the doctests in module module1. How can I do this? I've read the python docs, but I'm not any closer to success, here. Adding the lines

import doctest
import module1
suite = doctest.DocTestSuite(module1)

doesn't work. unittest.main() searches through the current file scope and runs every test case it finds, right? But DocTestSuite produces a test suite. How do I get unittest.main() to run the additional cases in the suite? Or am I just confused and deluded??

Once again, I'd be grateful for any help anyone can offer.

Answer 1

If you struggle with the load_tests solution, read this.

Note 1 - don't skip regular test discovery:

From unittest docs:

If load_tests exists then discovery does not recurse into the package, load_tests is responsible for loading all tests in the package.

This is important. If we simply add load_tests in the tests folder it will load the doctests but skip all other tests that should be discovered.

My solution is to add a sub-package in the tests folder for doctests discovery:

├── packagename
│   ├── __init__.py
│   └── packagename.py
├── setup.py
└── tests
    ├── __init__.py
    ├── doctest
    │   └── __init__.py
    └── unittests
        ├── __init__.py
        ├── test_something_1.py
        └── test_something_2.py

And:

# tests/doctest/__init__.py

import doctest
from packagename import x

def load_tests(loader, tests, ignore):
    tests.addTests(doctest.DocTestSuite(x))
    return tests

unittest will find all the tests and doctests. No extra information is needed:

python -m unittest

Note 2 - don't miss code in load_tests:

It's easy to miss importing code in tests/doctest/__init__.py::load_tests, then this code won't be tested with doctest.

Note 3 - unittest will discover it, but VSCode will hate you:

VScode doesn’t support the load_tests (there is an open issue about it).

As a workaround, use these parameters in .vscode/settings.json to load tests only from unittests folder:

{
    "python.testing.unittestArgs": [
        "-v",
        "-s",
        "./tests/unittests",
        "-p",
        "test*.py"
    ],
    "python.testing.pytestEnabled": false,
    "python.testing.unittestEnabled": true
}

Note 4 - it's easier with pytest:

See Alexander Pacha's answer

Answer 2

First I tried accepted answer from Andrey, but at least when running in Python 3.10 and python -m unittest discover it has led to running the test from unittest twice. Then I tried to simplify it and use load_tests and to my surprise it worked very well:

So just write both load_tests and normal unittest tests in a single file and it works!

import doctest
import unittest

import my_module_with_doctests


class ts(unittest.TestCase):
    def test_null(self):
        self.assertTrue(False)


# No need in any other extra code here


# Load doctests as unittest, see https://docs.python.org/3/library/doctest.html#unittest-api
def load_tests(loader, tests, ignore):
    tests.addTests(doctest.DocTestSuite(my_module_with_doctests))
    return tests

Answer 3

I would recommend to use pytest --doctest-modules without any load_test protocol. You can simply add both the files or directories with your normal pytests and your modules with doctests to that pytest call.

pytest --doctest-modules path/to/pytest/unittests path/to/modules

It discovers and runs all doctests as well.

See https://docs.pytest.org/en/latest/doctest.html

Answer 4

An update to this old question: since Python version 2.7 there is the load_tests protocol and there is no longer a need to write custom code. It allows you to add a function load_tests(), which a test loader will execute to update its collection of unit tests for the current module.

Put a function like this in your code module to package the module's own doctests into a test suite for unittest:

def load_tests(loader, tests, ignore):
    tests.addTests(doctest.DocTestSuite())
    return tests

Or, put a function like this into your unit test module to add the doctests from another module (for example, package.code_module) into the tests suite which is already there:

def load_tests(loader, tests, ignore):
    tests.addTests(doctest.DocTestSuite(package.code_module))
    return tests

When unittest.TestLoader methods loadTestsFromModule(), loadTestsFromName() or discover() are used unittest uses a test suite including both unit tests and doctests.

Answer 5

This code will automatically run the doctests for all the modules in a package without needing to manually add a test suite for each module. This can be used with Tox.

import doctest
import glob
import os
import sys
if sys.version_info < (2,7,):
    import unittest2 as unittest
else:
    import unittest

import mypackage as source_package

def load_module_by_path(path):
    """Load a python module from its path.

    Parameters
    ----------
    path : str
        Path to the module source file.

    Returns
    -------
    mod : module
        Loaded module.
    """
    import imp

    module_file_basename = os.path.basename(path)
    module_name, ext = os.path.splitext(module_file_basename)
    mod = imp.load_source(module_name, path)
    return mod

def file_contains_doctests(path):
    """Scan a python source file to determine if it contains any doctest examples.

    Parameters
    ----------
    path : str
        Path to the module source file.

    Returns
    -------
    flag : bool
        True if the module source code contains doctest examples.
    """
    with open(path) as f:
        for line in f:
            if ">>>" in line:
                return True
    return False

def load_tests(loader, tests, pattern):
    """Run doctests for all modules"""
    source_dir = os.path.dirname(source_package.__path__[0])
    python_source_glob = os.path.join(source_dir, source_package.__name__, "*.py")
    python_source_files = glob.glob(python_source_glob)
    for python_source_file in python_source_files:
        if not file_contains_doctests(python_source_file):
            continue
        module = load_module_by_path(python_source_file)
        tests.addTests(doctest.DocTestSuite(module))
    return tests

Answer 6

In this code i combined unittests and doctests from imported module

import unittest


class ts(unittest.TestCase):
    def test_null(self):
        self.assertTrue(True)


class ts1(unittest.TestCase):
    def test_null(self):
        self.assertTrue(True)

testSuite = unittest.TestSuite()    
testSuite.addTests(unittest.makeSuite(ts))
testSuite.addTest(unittest.makeSuite(ts1))

import doctest
import my_module_with_doctests

testSuite.addTest(doctest.DocTestSuite(my_module_with_doctests))
unittest.TextTestRunner(verbosity = 2).run(testSuite)