CODEWITHSUNDEEP
testingintegration-testing
Alan Evangelista
Alan Evangelista

Reputation: 3163

Should integration tests be independent?

I'm writing integration tests for a web application using Selenium. In a past project, I have created a single test function which tested all the features of the app. An example in Python using unittest:

class TestWebUI(unittest.Testcase)
    def test_app(self):
        login("admin")
        testListX()
        x = testCreateX()
        y = testCreateY(x)
        testUpdateX(x)
        testUpdateY(y)
        testDeleteX(x)
        testDeleteY(y)
        testCreateZ()

        login("user")
        testListX()
        testListY()
        testCreateZ()

In this previous project, a developer should only submit a patch if the big monolithic test (and its subtests) passed. To be sure, the tests would also run in a server when any patch was submitted in a server; the results would be displayed in the patch review tool and a patch could only be approved if all tests passed. This worked fine.

I'm on a new project and considered replacing the monolithic integration tests with smaller tests. However, that will require adding setup code (and therefore will require extra execution time) for each test. For instance, the test code above would become something like:

class TestWebUI(unittest.TestCase):
    def test_list_x(self):
        login("admin")
        (testListX() code here)

    def test_create_x(self):
        login("admin")
        (testCreateX() code here)

    def test_delete_x(self):
        login("admin")
        (create dummy x)
        (testDeleteX() code here)

    def test_create_y(self):
        login("admin")
        (create dummy x)
        (testCreateY(x) code here)

What are the benefits of the latter approach, besides the possibility of running tests concurrently? Is it common to split integration tests into several tests and duplicate test setup for each one?

This question mentions specific technologies such as Selenium, Python and unittest, but it is independent from them.

Upvotes: 2

Views: 1140

Answers (1)

Mureinik
Mureinik

Reputation: 311393

Generally speaking, splitting a big test into several smaller test is considered the better practice, and has several benefits.

First, it makes it easier see what failed when you perform a breaking change. It's much easier to see that test_create_y failed than to have to dig through logs/error messages when test_app failed in order to understand that the creation of y is broken.

Second, having smaller tests that you can run individually can speed up development cycles. Suppose you broke the creation of x - you'd want to fix it, and rerun just the failing test (test_create_x) to make sure your fix is good instead of running the entire (long) test suite. Eventually, when you get a working fix you'd want to submit it and run the entire test suite with it before merging, but there's usually no need to do so until you have a fix ready. Of course, if you do want to run the entire suite for each intermediate attempt, nothing's stopping you.

Third, it allows you to detect multiple failures at once. Once a test fails an assertion, it does not continue. Suppose you broke the creation of x, y and z. With separate tests, you'd have three failing tests, and could easily see that all three of these functions were broken. With a monolithic test, you'd see x was broken, fix it, and rerun the test only to see that y is broken, and then repeat the same process with z.

And finally, you need to remember that these are integration tests, not unit tests, and you don't necessarily have to break the tests up into the smallest units possible. If you're concerned about the duplicate operations that would make the execution time longer, you can take an intermediate approach and group some tests (i.e., those that need the same setup) into one test method, e.g.:

class TestWebUI(unittest.Testcase)
    def test_x(self):
        login("admin")
        testListX()
        x = testCreateX()
        testUpdateX(x)
        testDeleteX(x)

    def test_y(self):
        # same concept as test_x...

Upvotes: 2

Related Questions