CODEWITHSUNDEEP
pythondjangounit-testingdjango-unittest
lajarre
lajarre

Reputation: 5162

Django tests - print all test failures for a series of n times the same test

I'd like that every assertion test in a TestCase is actually tested, even if the first one fails. In my situation, all the assertions are of the same nature.

Actually I have something that evaluates formulas written as Python objects (figure it as formulas written as strings to be eval'd). I'd like to do something like:

class MyTest(TestCase):
   def test_something(self):
       for id in ids:
           for expression in get_formulas(id):
                for variable in extract_variables(expression):
                    self.assertIn(variable, list_of_all_variables)

=> I want to see printed all of the variables that are not in the list_of_all_variables!

This is necessary for me to review all my so-called formulas and be able to correct errors.

Some more context:

I'm having a variable number of tests to perform (depending on a list of IDs written in a versioned data file) in one app.

To have a variable number of TestCase instances, I did write a base class (mixin), then build on-the-fly classes with the use of 3-args type function (that is creating classes).

This way, I have n tests, corresponding to the n different ids. It's a first step, but what I want is that each and every assertion in those tests gets tested and the corresponding assertion errors get printed.

Upvotes: 1

Views: 302

Answers (1)

lajarre
lajarre

Reputation: 5162

As referenced in the question Continuing in Python's unittest when an assertion fails, failing at assertion errors is the hardcoded behavior of the TestCase class.

So instead of changing it's behavior, I generated a lot of different test_... methods to my classes, in the following style:

from django.test import TestCase
from sys import modules


# The list of all objects against which the tests have to be performed
formids = [12,124,234]
# get_formulas returns a list of formulas I have to test independently, linked to a formid
formulas = {id: get_formulas(id) for id in formids}

current_module = sys.modules(__name__)

def test_formula_method(self, formula):
    # Does some assertions
    self.assertNotEqual(formula.id, 0)

for formid in formids:
    attrs = {'formid': formid}
    for f in formulas[formid]:
        # f=f so the 2nd arg to test_formula_method is staying local
        # and not overwritten by last one in loop
        attrs['test_formula_%s' % f.name] = lambda self, f=f: test_formula_method(self, f)

    klass_name = "TestForm%s" % formid
    klass = type(klass_name, (TestCase,), attrs)

    setattr(current_module, klass_name, klass)

Upvotes: 1

Related Questions