Tests pass when run individually but not when the whole test class run

Question

I have solved a topCoder problem for which all the tests pass when I run them on their own. Nonetheless when I run the whole test class some of them fail. Could you, please, help me identify the reason for this behaviour? Here is my class and my tests:

 package com.topcoder.div2.stage1;

import java.util.Arrays;

public class GameOfStones {
    private int iterations = 0;
    public int count(int[] stones){
        int result = checkEquality(stones);
        return result;
    }

    private int checkEquality(int[] stones){
        int count = 0;
        int sum = 0;
        for(int k = 0; k< stones.length;k++){
            sum += stones[k];
        }
        if(stones.length > 0) {
            for (int i = 0; i < sum; i++) {
                Arrays.sort(stones);
                if(stones[stones.length-1] != 3) {
                    int j = 0;
                    while (j < stones.length - 1) {
                        if (stones[j] == stones[j + 1]) {
                            count++;
                        }
                        j++;
                    }
                    if (count == stones.length - 1) {
                        return iterations;
                    }
                    stones[0] = stones[0] + 2;
                    stones[stones.length - 1] = stones[stones.length - 1] - 2;
                    iterations++;
                    count = 0;
                }
            }
        }
        return -1;
    }
}

Test:

package com.topcoder.div2.stage1;

import org.testng.annotations.Test;

import static org.testng.Assert.assertEquals;

public class GameOfStonesTest {
    private GameOfStones gameOfStones = new GameOfStones();

    @Test
    public void test1() {
        int expected = 0;
        int[] given = {17};

        int actual = gameOfStones.count(given);

        assertEquals(expected, actual);
    }

    @Test
    public void test2() {
        int expected = 3;
        int[] given ={7, 15, 9, 5};

        int actual = gameOfStones.count(given);
        assertEquals(actual, expected);
    }

    @Test
    public void test3() {
        int expected = -1;
        int[] given ={2, 8, 4};

        int actual = gameOfStones.count(given);
        assertEquals(actual, expected);
    }

    @Test
    public void test4() {
        int expected = -1;
        int[] given ={10, 15, 20, 12, 1, 20};

        int actual = gameOfStones.count(given);
        assertEquals(actual, expected);
    }

    @Test
    public void test5(){
        int expected = 277;
        int[] given ={17, 1, 27, 29, 13, 1, 27, 3, 19, 3, 25, 1, 11, 9, 7, 17, 31, 25, 5, 11, 31, 9,
                15, 3, 3, 3, 11, 11, 1, 41, 5, 95, 7, 3, 41, 31, 7, 13, 15, 5, 17, 3, 9, 3, 11,
                27, 1, 23, 15, 5, 43, 11, 17, 7, 1, 3, 13, 69, 3, 43, 21, 1, 25, 1, 3, 11, 5, 43,
                13, 7, 15, 1, 1, 55, 37, 9, 5, 7, 21, 3, 23, 15, 1, 9, 3, 35, 13, 17, 7, 17, 27, 5,
                9, 19, 13, 1, 1, 1, 29};
        int actual = gameOfStones.count(given);
        assertEquals(actual, expected);
    }

    @Test
     public void test6(){
        int expected = 539;
        int[] given ={1, 29, 11, 35, 57, 15, 85, 19, 5, 47, 53, 5, 63, 19, 13, 63, 27, 43, 53, 75, 67, 93, 33, 31, 47, 3,
                63, 17, 11, 53, 35, 23, 17, 45, 31, 19, 63, 75, 5, 3, 49, 19, 11, 89, 21, 69,
                71, 5, 45, 81, 31, 13, 11, 19, 7, 99, 33, 63, 19, 57, 73, 29, 35, 9, 47,
                1, 17, 7, 13, 31, 5, 85, 95, 23, 45, 65, 63, 41, 81, 33, 45, 1, 15,
                45, 19, 87, 51, 7, 13, 39, 1, 59, 29, 35, 1, 43};
        int actual = gameOfStones.count(given);
        assertEquals(actual, expected);
    }

    @Test
    public void test7() {
        int expected = 0;
        int[] given ={100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100,
                100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100,
                100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100,
                100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100,
                100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100,
                100, 100};
        int actual = gameOfStones.count(given);
        assertEquals(actual, expected);
    }

    @Test
    public void test8() {
        int expected = 11;
        int[] given ={3, 5, 21, 31};
        int actual = gameOfStones.count(given);
        assertEquals(actual, expected);
    }

    @Test
    public void test9() {
        int expected = 13;
        int[] given ={44, 6, 46};
        int actual = gameOfStones.count(given);
        assertEquals(actual, expected);
    }

}

P.S if you know any suggestions of improving the solution you are more than welcome to include them in your answer.

Answer 1

Late on this but had the same behavior, it could be the versions like it was stated by Udara

For me intellij ran without issue but pipeline failed, ended up being that at least on my local the tests were being "cleaned up" but pipeline doesn't so mocks for a class where affecting another un-related tests.

I used this other post to clean them, not sure if is the best solution but it works

Answer 2

You can also try @BeforeEach for Junit5

@BeforeEach
public void setup() {
  // Some code here
}

Answer 3

I would like to add my experience regarding this issue.

After doing many debuggins I could confirm that there is no garbage data in the DB and also no static variables effects to the functionality.

And also the same unit tests, pass individually and all at once as expected in another environment.

Java versions, MySQL versions are the same in both environments. The only difference was the maven version. So, I configured the same maven version which was used in the successful environment.

Then after issue solved.

Answer 4

Since I am opening the same activity under test in many test cases executed one after another and same IntentsTestRule class activity

I could resolve the issue in my case by calling finishActivity on activityTestRule class and intentTestRule class

eg:

@After
fun method() {

mainActivityTestRule.finishActivity()

mIntentsRule.finishActivity()

}

Answer 5

I had same issue. I needed to mock a logger, which was a static field. So eventually class loader creates only a single instance of the static field during the first invocation of a class under the test and disregards all further mocking and stubbing. When run separately, test was green, because the logger was initialized and loaded as expected, but when run all together, with other test methods, it got initialized as a concrete object, not a mock. Workaround:

• create @BeforeClass method to ensure that the right instance of static field will be created in the first place:

    @BeforeClass
    public static void setupBeforeClass() {
      PowerMockito.mockStatic(LoggerFactory.class);
      loggerMock = mock(Logger.class);
      when(LoggerFactory.getLogger(any(Class.class))).thenReturn(loggerMock);
   }

• Interactions on the mock are getting accumulated from different test executions. Therefore, to be sure that you get a clean instance of the mock on each test method execution, reset the mock whether in @Before or @After method:

      @Before
      public void setup() {

        // Reset interactions on the mocked logger
        Mockito.reset(loggerMock);

      }

Note, that in my example I used PowerMock so you need a corresponding runner @RunWith(PowerMockRunner.class) and @PrepareForTest({LoggerFactory.class, MyClass.class)} statements.

Answer 6

You are sharing a single instance of the class under test across all tests. I'd remove the initial assignment and add this:

private GameOfStones gameOfStones; // Don't create an instance here

@BeforeMethod
public void setUp() {
    gameOfStones = new GameOfStones();
}

... which will use a new instance for each test. Good practice would also be to clean up after each test:

@AfterMethod
public void tearDown() {
    gameOfStones = null;
}

In the example given here, fixing the class scoped variable causing the problem to be method scoped instead would also fix the issue, but as the software under test gets more complex it's good to start doing proper test set up and tear down.