CODEWITHSUNDEEP
c++unit-testingcmake
user1291510
user1291510

Reputation: 265

CMake testing causing error when tests fail

I've been looking around for a solution to this but cannot seem to find out why this happens. I use CMake for a small project and I wanted to add unit tests cmakes add_test. It works well when tests are passing but when they fail, I get an odd error message that I cannot interpret.

I've made a minimal example to illustrate what happens.

CMakeLists.txt

cmake_minimum_required (VERSION 2.6)

project (TestExample)

enable_testing()

add_executable(succeedExample succeed.cpp)
add_executable(failExample fail.cpp)

add_test(NAME succeedTest COMMAND succeedExample)
add_test(NAME failTest COMMAND failExample)

succeed.cpp

int main()
{
    return 0;
}

fail.cpp

int main()
{
    return 1;
}

I use Visual Studio 2015, and when building the RUN_TESTS target, I get the following error:

Severity    Code    Description Project File    Line    Suppression State
Error   MSB3073 The command "setlocal
"C:\Program Files (x86)\CMake\bin\ctest.exe" --force-new-ctest-process -C Debug
if %errorlevel% neq 0 goto :cmEnd
:cmEnd
endlocal & call :cmErrorLevel %errorlevel% & goto :cmDone
:cmErrorLevel
exit /b %1
:cmDone
if %errorlevel% neq 0 goto :VCEnd
:VCEnd" exited with code 8. RUN_TESTS   C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Microsoft.CppCommon.targets  133

Yet if I change the fail.cpp to return 0; I get

1>------ Build started: Project: RUN_TESTS, Configuration: Debug Win32 ------
1>  Test project C:/Users/.../projects/cmaketestexample/build
1>      Start 1: succeedTest
1>  1/2 Test #1: succeedTest ......................   Passed    0.01 sec
1>      Start 2: failTest
1>  2/2 Test #2: failTest .........................   Passed    0.02 sec
1>
1>  100% tests passed, 0 tests failed out of 2
1>
1>  Total Test time (real) =   0.03 sec
========== Build: 1 succeeded, 0 failed, 1 up-to-date, 0 skipped ==========

Can anyone point me towards a solution to fix this? Am I missing something obvious?

Thanks :)

EDIT

Visual studio version is 14.0 (Visual Studio 2015 community)

Cmake 3.5.2 - yet I set the version to 2.6 in the CMakeLists.txt.

EDIT2

running ctest -C Debug on command line:

Test project C:/Users/.../projects/cmaketestexample/build
    Start 1: succeedTest
1/2 Test #1: succeedTest ......................   Passed    0.02 sec
    Start 2: failTest
2/2 Test #2: failTest .........................***Failed    0.02 sec

50% tests passed, 1 tests failed out of 2

Total Test time (real) =   0.07 sec

The following tests FAILED:
          2 - failTest (Failed)
Errors while running CTest

Changing fail.cpp to

#include <iostream>
int main()
{
    std::cout << "Does it work now?" << std::endl;
    return 1;
}

makes no difference.

EDIT3

A similar error occur on Ubuntu 14.04.4 LTS using

Upvotes: 3

Views: 8886

Answers (1)

Florian
Florian

Reputation: 42862

I've given you code a try and what you see is the little script that CMake embeds the ctest call into. And ctest does fail - return a non-zero value - if any of tests it is executing does fail. And this behavior of ctest is by intention to stop any scripts/builds running the tests if encountering errors.

The possible error return codes of ctest are a combination of the following bits:

// provide some more detailed info on the return code for ctest
enum {
  UPDATE_ERRORS    = 0x01,
  CONFIGURE_ERRORS = 0x02,
  BUILD_ERRORS     = 0x04,
  TEST_ERRORS      = 0x08,
  MEMORY_ERRORS    = 0x10,
  COVERAGE_ERRORS  = 0x20,
  SUBMIT_ERRORS    = 0x40
};

So I see the following options:

  1. Ignore the strange error, the output you're looking for is just above the error message: 

    1>------ Build started: Project: RUN_TESTS, Configuration: Debug Win32 ------
1>  Test project C:/temp/StackOverflow/TestError/VS2015
1>      Start 1: succeedTest
1>  1/2 Test #1: succeedTest ......................   Passed    0.03 sec
1>      Start 2: failTest
1>  2/2 Test #2: failTest .........................***Failed    0.03 sec
1>
1>  50% tests passed, 1 tests failed out of 2
1>
1>  Total Test time (real) =   0.10 sec
1>
1>  The following tests FAILED:
1>        2 - failTest (Failed)
1>  Errors while running CTest
1>C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Microsoft.CppCommon.targets(133,5): error MSB3073: The command "setlocal
1>C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Microsoft.CppCommon.targets(133,5): error MSB3073: "C:\Program Files\CMake\bin\ctest.exe" --force-new-ctest-process -C Debug
1>C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Microsoft.CppCommon.targets(133,5): error MSB3073: if %errorlevel% neq 0 goto :cmEnd
1>C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Microsoft.CppCommon.targets(133,5): error MSB3073: :cmEnd
1>C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Microsoft.CppCommon.targets(133,5): error MSB3073: endlocal & call :cmErrorLevel %errorlevel% & goto :cmDone
1>C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Microsoft.CppCommon.targets(133,5): error MSB3073: :cmErrorLevel
1>C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Microsoft.CppCommon.targets(133,5): error MSB3073: exit /b %1
1>C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Microsoft.CppCommon.targets(133,5): error MSB3073: :cmDone
1>C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Microsoft.CppCommon.targets(133,5): error MSB3073: if %errorlevel% neq 0 goto :VCEnd
1>C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Microsoft.CppCommon.targets(133,5): error MSB3073: :VCEnd" exited with code 8.
========== Build: 0 succeeded, 1 failed, 1 up-to-date, 0 skipped ==========

  2. Add your own custom target calling ctest on VS without a script, giving just one error:

    if (CMAKE_CONFIGURATION_TYPES)
    add_custom_target(
        RUN_CTEST
            COMMAND ${CMAKE_CTEST_COMMAND} -C $<CONFIGURATION> --force-new-ctest-process 
            WORKING_DIRECTORY "${CMAKE_BINARY_DIR}"
    )
endif()

  3. Add your own custom target command that does ignore the return value by moving the ctest call into a separate script (see references).

References

Upvotes: 2

Related Questions