Reputation: 1954
I am writing a command-line utility that when given invalid args should print an error message and return with non-zero exit code. I want to write a test that checks this behaviour.
The details of this utility does not matter, so let me use cat nonexistent
command to illustrate the issue:
> cat nonexistent
cat: nonexistent: No such file or directory
Let's write a test (CMakeLists.txt):
cmake_minimum_required(VERSION 3.15)
project(failing-test)
enable_testing()
add_test(NAME failing-test COMMAND cat nonexistent)
mkdir build
cd build
cmake ..
cmake --build .
ctest
Output:
Test project /home/failing-test/build
Start 1: failing-test
1/1 Test #1: failing-test .....................***Failed 0.00 sec
0% tests passed, 1 tests failed out of 1
Total Test time (real) = 0.00 sec
The following tests FAILED:
1 - failing-test (Failed)
Errors while running CTest
Output from these tests are in: /home/failing-test/build/Testing/Temporary/LastTest.log
Use "--rerun-failed --output-on-failure" to re-run the failed cases verbosely.
OK, the command fails as it should, but I want this failure to pass the test.
If I now add WILL_FAIL
property:
cmake_minimum_required(VERSION 3.15)
project(failing-test)
enable_testing()
add_test(NAME failing-test COMMAND cat nonexistent)
set_property(TEST failing-test PROPERTY WILL_FAIL true)
the test passes. All good, but I also want to ensure the command outputs the expected message:
cmake_minimum_required(VERSION 3.15)
project(failing-test)
enable_testing()
add_test(NAME failing-test COMMAND cat nonexistent)
set_property(TEST failing-test PROPERTY WILL_FAIL true)
set_property(TEST failing-test PROPERTY PASS_REGULAR_EXPRESSION "cat: nonexistent: No such file or directory")
Result:
Test project /home/failing-test/build
Start 1: failing-test
1/1 Test #1: failing-test .....................***Failed Required regular expression found. Regex=[cat: nonexistent: No such file or directory] 0.00 sec
0% tests passed, 1 tests failed out of 1
Total Test time (real) = 0.00 sec
The following tests FAILED:
1 - failing-test (Failed)
Errors while running CTest
Output from these tests are in: /home/failing-test/build/Testing/Temporary/LastTest.log
Use "--rerun-failed --output-on-failure" to re-run the failed cases verbosely.
No good. If I remove the line with WILL_FAIL
property, the test passes again, even though the commands return non-zero exit code.
Is there a way to have both checks in a single test, or do I need to test non-zero exit code and message in separate tests?
Upvotes: 0
Views: 19