CODEWITHSUNDEEP
pythonunit-testingpytest
crystalattice
crystalattice

Reputation: 5089

With pytest, why are single test results different from running all tests?

Using pytest with Python 3.6, I'm making a piping simulation for a project. I have tests to work with different valve alignments and the like.

The weird thing is, if I run all tests within a directory, all the tests pass. If I try to run a single test class, it fails. I've tried adding setup/teardown methods to separate it from the rest of the code, in case it was somehow being affected by the other classes.

Here is the test code (ffc is the component library, fff is the method library):

class TestGate3TankLevels:
    @classmethod
    def setup_class(cls):
        ffc.tank2.level = 18.0
        fff.change_tank_level(ffc.tank2, ffc.tank2.level)
        fff.gate1_open()
        fff.gate2_open()
        fff.gate3_open()
        fff.gate4_open()

    def test_gate3_tank_levels(self):
        assert ffc.gate1.position == 100
        assert ffc.gate1.flow_out == 19542.86939891452
        assert ffc.gate1.press_out == 13.109851301499999

        assert ffc.gate2.position == 100
        assert ffc.gate2.flow_out == 19542.86939891452
        assert ffc.gate2.press_out == 6.5549256507499996

        assert ffc.gate3.position == 100
        assert ffc.gate3.flow_in == 19542.86939891452
        assert ffc.gate3.press_in == 13.109851301499999
        assert ffc.gate3.flow_out == 19542.86939891452
        assert ffc.gate3.press_out == 13.109851301499999
        assert ffc.gate6.flow_in == 19542.86939891452
        assert ffc.gate6.press_in == 13.109851301499999

        assert ffc.gate4.position == 100
        assert ffc.gate4.flow_in == 0.0
        assert ffc.gate4.press_in == 0.0
        assert ffc.gate4.flow_out == 0.0
        assert ffc.gate4.press_out == 0.0

    @classmethod
    def teardown_class(cls):
        ffc.tank2.level = 32.0
        fff.change_tank_level(ffc.tank2, ffc.tank2.level)

When ran by itself, I get the following results:

F
test_fuel_components.py:408 (TestGate3TankLevels.test_gate3_tank_levels)
19542.86939891452 != 39085.73879782904

Expected :39085.73879782904
Actual   :19542.86939891452
 <Click to see difference>

self = <VirtualPLC.tests.models.fuel_farm.test_fuel_components.TestGate3TankLevels object at 0x7f9b35289a20>

    def test_gate3_tank_levels(self):
        assert ffc.gate1.position == 100
        assert ffc.gate1.flow_out == 19542.86939891452
        assert ffc.gate1.press_out == 13.109851301499999

        assert ffc.gate2.position == 100
        assert ffc.gate2.flow_out == 19542.86939891452
        assert ffc.gate2.press_out == 6.5549256507499996

        assert ffc.gate3.position == 100
        assert ffc.gate3.flow_in == 19542.86939891452
        assert ffc.gate3.press_in == 13.109851301499999
        assert ffc.gate3.flow_out == 19542.86939891452
        assert ffc.gate3.press_out == 13.109851301499999
>       assert ffc.gate6.flow_in == 19542.86939891452
E       assert 39085.73879782904 == 19542.86939891452
E        +  where 39085.73879782904 = <PipingSystems.valve.valve.Gate object at 0x7f9b3515ce80>.flow_in
E        +    where <PipingSystems.valve.valve.Gate object at 0x7f9b3515ce80> = ffc.gate6

test_fuel_components.py:423: AssertionError

However, if I run all the tests within the test_fuel_components.py file, I everything passes:

Testing started at 2:16 PM ...
/home/cody/PycharmProjects/VirtualPLC/venv/bin/python /home/cody/.local/share/JetBrains/Toolbox/apps/PyCharm-P/ch-0/182.3341.8/helpers/pycharm/_jb_pytest_runner.py --path /home/cody/PycharmProjects/VirtualPLC/tests/models/fuel_farm/test_fuel_components.py
Launching py.test with arguments /home/cody/PycharmProjects/VirtualPLC/tests/models/fuel_farm/test_fuel_components.py in /home/cody/PycharmProjects/VirtualPLC/tests/models/fuel_farm

============================= test session starts ==============================
platform linux -- Python 3.6.3, pytest-3.6.2, py-1.5.3, pluggy-0.6.0
rootdir: /home/cody/PycharmProjects/VirtualPLC/tests/models/fuel_farm, inifile:collected 20 items

test_fuel_components.py ....................                             [100%]

========================== 20 passed in 0.16 seconds ===========================
Process finished with exit code 0

If I move this test class to another location in the file (right now it is at the end of all the other tests), it causes any subsequent tests to fail, as well as failing itself.

I don't understand how a test class can fail by itself but is fine when ran with other tests.

Edit Link to other tests (too long to post here)

Upvotes: 4

Views: 3576

Answers (2)

crystalattice
crystalattice

Reputation: 5089

The global state of the storage tanks was the problem. I had tests at the beginning that changed the level of one of the tanks but never returned it to its normal state.

By adding some function calls prior to starting the rest of the tests and ensuring that the tank levels were "full", it fixed all the problems: 

    fff.change_tank_level(ffc.tank1, 36)
    fff.change_tank_level(ffc.tank2, 36)

where 36 is a full tank.

So, when I get to the TestGate3TankLevels class, I changing the tank levels works correctly. The teardown method then returns the tank levels back to normal.

Upvotes: 0

Matthew Story
Matthew Story

Reputation: 3783

So the problem here is that the state of each tank and valve is initialized once when the tank models are imported, like tank1 here:

# Storage tanks
# Assumes 36 ft tall tank w/ 1 million gallon capacity = 27778 gallons per foot
# Assumes 16 inch diam transfer piping
tank1 = tank.Tank("Tank 1", level=36.0, fluid_density=DENSITY, spec_gravity=SPEC_GRAVITY, outlet_diam=16,
                  outlet_slope=0.25)
tank1.static_tank_press = tank1.level
tank1.gravity_flow(tank1.pipe_diam, tank1.pipe_slope, tank1.pipe_coeff)

It's hard to offer a specific critique on this design without knowing more about what this project is supposed to do, but this design has a huge amount of global state which makes testing extremely difficult.

Regardless of the existing design's pros and cons, what you need to do is to reset the state of each tank to a known state at the beginning of every test if you want the tests to be reliable and isolated. Given how much global state is involved here your options are pretty limited.

My recommendation when you have this much state stored globally would be to reset back to a known state after every test by using the setUp and tearDown methods.

This is untested, but you're looking for something like this:

original_states = {}
tanks = [tank1, tank2, tank3, tank4, tank5, tank6]
tank_attrs = [ "name", "level", "fluid_density", "spec_grav",
               "tank_press", "flow_out", "pipe_diam", "pipe_slope" ]
def setUp(self):
    for tank in self.tanks:
        self.original_states[id(tank)] = {}
        for attr in self.tank_attrs:
            self.original_states[id(tank)] = getattr(tank, attr)

def tearDown(self):
    try:
        for tank in self.tanks:
            for attr, value in self.original_states[id(tank)].items():
                setattr(tank, attr, value)
    finally:
        self.original_states = {}

Which basically stores the original state before each test and then restores it following each test. There is some magic around some of these attributes so it might take a little bit of trial and error to get it to work 100%.

Upvotes: 2

Related Questions