Python 3 unittest: How to extract results of tests?

Question

I am using Python's (3.4.1) unittest module for my unit tests.

I load all my testing module files using imports and then run unittest.main():

import unittest
import testing_module1
import testing_module2
# [...]
if __name__ == '__main__':
    unittest.main()

This works perfectly for me as it is simple and respect the command line arguments I use to control verbosity or which test(s) to run.

I want to continue to output the same information, but I would like to generate an XML file from the results. I tried xmlrunner (https://github.com/xmlrunner/unittest-xml-reporting/) but:

• it does not output as much info to stdout as the standard runner;
• it uses a specific format of the XML that doesn't suites me.

I would like to generate the XML (I don't mind doing it manually) with the format I need but with minimal change to how the tests are run.

What are my options?

1. I could write my own TestRunner but I don't want to re-write everything, I just want to add extra output to the actual runner with minimal code change.
2. I could inherit unittest.TextTestRunner but I fear that adding XML output to it would require re-writing every methods, loosing the advantage of inheritance in the first place.
3. I could try to extract the test results after the call to unittest.main() and parse it. The problem here is that unittest.main() seems to exit when it's done so any code after it is not executed.

Any suggestion?

Thanks!

Answer 1

I have faced the same issue with catching FAIL events from unittest lib. Following big_gie's answer, this code appeared:

File testFileName_1.py

import unittest

class TestClassToTestSth(unittest.TestCase):
    def test_One(self):
        self.AssertEqual(True, False, 'Hello world')

import unittest
from io import StringIO
import testFileName_1

def suites():
    return [
        # your testCase classes, for example
        testFileName_1.TestClassToTestSth,
        testFileName_445.TestClassToTestSomethingElse,
    ]

class TextTestResult(unittest.TextTestResult):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.slack = Slack('data-engineering-tests')

    def addFailure(self, test, err):
        super().addFailure(test, err)

        # Whatever you want here
        print(err, test)
        print(self.failures)


class TextTestRunner(unittest.TextTestRunner):
    resultclass = TextTestResult
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)


loader = unittest.TestLoader()
suite = unittest.TestSuite()

stream = StringIO()
for test_case in suites():
    suite.addTests(loader.loadTestsFromTestCase(test_case))
runner = TextTestRunner(stream=stream)
result = runner.run(suite)

stream.seek(0)
print(stream.read())

Answer 2

I ended up writing two new classes, inheriting from unittest.TextTestResult and unittest.TextTestRunner. That way, I could run main like that:

unittest.main(testRunner=xmlrunner.XMLTestRunner(...))

I overloaded unittest.TextTestRunner's __init__ and those from unittest.TextTestResult:

• addSuccess()
• addError()
• addFailure()
• addSubTest()

For example:

def addSuccess(self, test):
    super().addSuccess(test)
    [... store the test into list, dictionary, whatever... ]

Since these add*() functions are called with the actual test, I can store them in a global list and parse them at the end of my XMLTestRunner.run():

def run(self, test):
    result = super().run(test)
    self.save_xml_report(result)
    return result

Note that these functions are normally defined in /usr/lib/python3.4/unittest/runner.py.

Warning note: By using an actual object passed unittest.main()'s testRunner argument as shown, the command line arguments given when launching python are ignored. For example, increasing verbose level with -v argument is ignored. This is because the TestProgram class, defined in /usr/lib/python3.4/unittest/main.py, detects if unittest.main() was run with testRunner being a class or an object (see runTests() near the end of the file). If you give just a class like that:

unittest.main(testRunner=xmlrunner.XMLTestRunner)

then command line arguments are parsed. But you pass an instantiated object (like I need), runTests() will just use it as is. I thus had to parse arguments myself in my XMLTestRunner.__init__():

# Similar to what /usr/lib/python3.4/unittest/main.py's TestProgram._getParentArgParser() does.
import argparse
parser = argparse.ArgumentParser(add_help=False)
parser.add_argument('-v', '--verbose', dest='verbosity',
                    action='store_const', const=2, default=1,  # Add default=1, not present in _getParentArgParser()
                    help='Verbose output')
parser.add_argument('-q', '--quiet', dest='verbosity',
                    action='store_const', const=0,
                    help='Quiet output')
parser.add_argument('-f', '--failfast', dest='failfast',
                    action='store_true',
                    help='Stop on first fail or error')
parser.add_argument('-c', '--catch', dest='catchbreak',
                    action='store_true',
                    help='Catch ctrl-C and display results so far')
parser.add_argument('-b', '--buffer', dest='buffer',
                    action='store_true',
                    help='Buffer stdout and stderr during tests')

Answer 3

How does this work for you. Capture the output of unittest, which goes to sys.stderr, in a StringIO. Continue after unittest.main by adding `exit=False'. Read the captured output and process as you want. Proof of concept:

import contextlib
import io
import sys
import unittest

class Mytest(unittest.TestCase):
    def test_true(self):
        self.assertTrue(True)

@contextlib.contextmanager
def err_to(file):
    old_err = sys.stderr
    sys.stderr = file
    yield
    sys.stderr = old_err

if __name__ == '__main__':
    result = io.StringIO()
    with err_to(result):
        unittest.main(exit=False)
    result.seek(0)
    print(result.read())

This prints (to sys.stdout)

----------------------------------------------------------------------
Ran 1 test in 0.000s

OK

Note: contextlib has redirect_stdout, but not redirect_stderr. The above is simpler that the contextlib code. The above assumes that there are no exceptions not caught by unittest. See the contextlib.contextmanager doc for adding try: except: finally. I leave that to you.