Krzysiek Karbowiak
Krzysiek Karbowiak

Reputation: 1954

How to check that a program fails with message

I am writing a command-line utility that when given invalid args should print an error message and return with non-zero exit code. I want to write a test that checks this behaviour.

The details of this utility does not matter, so let me use cat nonexistent command to illustrate the issue:

> cat nonexistent
cat: nonexistent: No such file or directory

Let's write a test (CMakeLists.txt):

cmake_minimum_required(VERSION 3.15)

project(failing-test)

enable_testing()

add_test(NAME failing-test COMMAND cat nonexistent)
mkdir build
cd build
cmake ..
cmake --build .
ctest

Output:

Test project /home/failing-test/build
    Start 1: failing-test
1/1 Test #1: failing-test .....................***Failed    0.00 sec

0% tests passed, 1 tests failed out of 1

Total Test time (real) =   0.00 sec

The following tests FAILED:
          1 - failing-test (Failed)
Errors while running CTest
Output from these tests are in: /home/failing-test/build/Testing/Temporary/LastTest.log
Use "--rerun-failed --output-on-failure" to re-run the failed cases verbosely.

OK, the command fails as it should, but I want this failure to pass the test.

If I now add WILL_FAIL property:

cmake_minimum_required(VERSION 3.15)

project(failing-test)

enable_testing()

add_test(NAME failing-test COMMAND cat nonexistent)
set_property(TEST failing-test PROPERTY WILL_FAIL true)

the test passes. All good, but I also want to ensure the command outputs the expected message:

cmake_minimum_required(VERSION 3.15)

project(failing-test)

enable_testing()

add_test(NAME failing-test COMMAND cat nonexistent)
set_property(TEST failing-test PROPERTY WILL_FAIL true)
set_property(TEST failing-test PROPERTY PASS_REGULAR_EXPRESSION "cat: nonexistent: No such file or directory")

Result:

Test project /home/failing-test/build
    Start 1: failing-test
1/1 Test #1: failing-test .....................***Failed  Required regular expression found. Regex=[cat: nonexistent: No such file or directory]  0.00 sec

0% tests passed, 1 tests failed out of 1

Total Test time (real) =   0.00 sec

The following tests FAILED:
          1 - failing-test (Failed)
Errors while running CTest
Output from these tests are in: /home/failing-test/build/Testing/Temporary/LastTest.log
Use "--rerun-failed --output-on-failure" to re-run the failed cases verbosely.

No good. If I remove the line with WILL_FAIL property, the test passes again, even though the commands return non-zero exit code.

Is there a way to have both checks in a single test, or do I need to test non-zero exit code and message in separate tests?

Upvotes: 0

Views: 19

Answers (0)

Related Questions