C++ Unit Test fails when running all tests or after a certain test

Question

I am doing unit tests with GoogleTest, and today when I added one new test, I have found that it passes when running alone and fails when running all tests. I have read some questions and answers, but they are linked to C#.

I use C++ and CMake for doing this.

My CMakeLists.txt looks like this:

cmake_minimum_required(VERSION 2.8)

option(test "Build all tests." OFF)

set(EXECUTABLE_NAME proj)

project(${EXECUTABLE_NAME})

set(CMAKE_CXX_FLAGS "-g -Wall")

include_directories(src/main/cpp
                    ${Boost_INCLUDE_DIRS}
                    )

find_package(OpenCV REQUIRED core
                             imgproc
                             highgui
                             )
find_package(Boost REQUIRED COMPONENTS filesystem 
                                       system 
                                       regex 
                                       program_options
                                       )


  add_executable( ${EXECUTABLE_NAME}
# .........
                         )

  target_link_libraries(${EXECUTABLE_NAME} ${OpenCV_LIBRARIES}
                                           ${Boost_LIBRARIES}
                                           )

if (test)
  find_package(GTest REQUIRED)

  enable_testing()

  include_directories( ${GTEST_INCLUDE_DIRS} )

  add_executable(${EXECUTABLE_NAME}_test1
#  .............
            )

  add_executable(${EXECUTABLE_NAME}_test2
#  .............
            )  
  add_executable(${EXECUTABLE_NAME}_test3
#  .............
            )  
  add_executable(${EXECUTABLE_NAME}_test4
#  .............
            )

  target_link_libraries(${EXECUTABLE_NAME}_test1
                    ${OpenCV_LIBRARIES}
                    ${Boost_LIBRARIES}
                    ${GTEST_LIBRARIES}
                    gtest_main
                    pthread
                    )

  target_link_libraries(${EXECUTABLE_NAME}_test1
                    ${OpenCV_LIBRARIES}
                    ${Boost_LIBRARIES}
                    ${GTEST_LIBRARIES}
                    gtest_main
                    pthread
                    )
  target_link_libraries(${EXECUTABLE_NAME}_test2
                    ${OpenCV_LIBRARIES}
                    ${Boost_LIBRARIES}
                    ${GTEST_LIBRARIES}
                    gtest_main
                    pthread
                    )
  target_link_libraries(${EXECUTABLE_NAME}_test3
                    ${OpenCV_LIBRARIES}
                    ${Boost_LIBRARIES}
                    ${GTEST_LIBRARIES}
                    gtest_main
                    pthread
                    )
  target_link_libraries(${EXECUTABLE_NAME}_test4
                    ${OpenCV_LIBRARIES}
                    ${Boost_LIBRARIES}
                    ${GTEST_LIBRARIES}
                    gtest_main
                    pthread
                    )

  add_test(${EXECUTABLE_NAME}_test1
       ${EXECUTABLE_NAME}_test1
       )

  add_test(${EXECUTABLE_NAME}_test2
       ${EXECUTABLE_NAME}_test2
       )  
  add_test(${EXECUTABLE_NAME}_test3
       ${EXECUTABLE_NAME}_test3
       )  
  add_test(${EXECUTABLE_NAME}_test4
       ${EXECUTABLE_NAME}_test4
       )
endif()

I use a test like this:

#pragma once

#include <gtest/gtest.h>

#include "My.hpp"

/** Testing 
 */
TEST(qsdf, asdf)
{
 // ... initialization
 // ... test
}

I have tried to use TEST_F instead of TEST but it crashes, it says that qsdf was not declared (I have to admit I have not done many unit tests before).

I have read that there is the possibility that the other test modify something and I think it is true, because it I run the test individually after some of the others it fails again.

My question is: How to initialize each test in C++ using CMake (under Linux, if this matters)?

Answer 1

It definitely sounds like a problem involving a shared resource (as you suspect).

Initialization of each test is done completely from within your test file, without any extra involvement from CMake, etc. As you mentioned, tests usually consist of an initialization phase followed by the actual test. Additionally, it's not unusual to include a cleanup step that takes place after the test completes. For example:

TEST(qsdf, asdf)
{
    // ... initialization
    // ... test
    // ... **cleanup**
}

Typically, the initialization and cleanup phases just involve allocating and deallocating variables used in your test. But those phases are generally about making preparations to run your test, and cleaning up after it.

If you are testing code that shares resources, then the initialization phase consists of initializing those resources as needed; the cleanup phase consists of resetting those resources back to the state they were in before the initialization phase.

When you have tests that share resources, it's helpful to use a test fixture to coordinate the initialization/cleanup of the resources. This is the difference between the TEST and TEST_F macros in googletest - the latter represents a test that is based on a test fixture.

The googletest AdvancedGuide has a lot of information about using test fixtures:

https://github.com/google/googletest/blob/master/googletest/docs/advanced.md#sharing-resources-between-tests-in-the-same-test-case

The reason your code broke when you changed TEST to TEST_F is because you didn't define the test fixture that the test was expecting to find, which is the first parameter to the macro. In googletest, a test fixture is just a class that inherits from the ::testing::Test class, so you would need to do something like this:

#pragma once

#include <gtest/gtest.h>

#include "My.hpp"

class MyTestFixture : public ::testing::Test
{
public:
    virtual void SetUp() // This is defined in ::gtest::Test and overridden here
    {
        // This code will run before every test that uses this fixture
    }

    virtual void TearDown() // This is defined in ::gtest::Test and overridden here
    {
        // This code will run after every test that uses this fixture
    }
};

/** Testing 
 */
TEST_F(MyTestFixture, asdf)
{
    // ... initialization (after SetUp)
    // ... test
    // ... cleanup (before TearDown)
}