CODEWITHSUNDEEP
command-linecmakectest
jlconlin
jlconlin

Reputation: 15054

How to pass command-line arguments in CTest at runtime

I'm using CTest and want to pass command-line arguments to the underlying tests at runtime. I know there are ways to hard code command-line arguments into the CMake/CTest script, but I want to specify the command-line arguments at runtime and have those arguments passed through CTest to the underlying test.

Is this even possible?

Upvotes: 31

Views: 15664

Answers (4)

Sergey Bronnikov
Sergey Bronnikov

Reputation: 69

I had a similar problem with passing a number of test runs to the test. If environment variable is set, then use value, otherwise use constant default value.

The trick is to use shell for running a test executable and use parameter expansion.

With

a=${VAR:-20}

shell will assign to "a" the value of VAR if VAR is set, otherwise it will assign it the default value 20 -- this can also be a result of an expression.

Snippet with CMake code is the following:

find_program(SHELL sh)

string(JOIN " " LIBFUZZER_OPTS
    -mutate_depth=20
    -runs=$\{RUNS:-${DEFAULT_RUNS_NUMBER}\}
    -artifact_prefix=${test_name}_
)
add_test(NAME ${test_name}
         COMMAND ${SHELL} -c "$<TARGET_FILE:${test_name}> ${LIBFUZZER_OPTS}"
)

Running a test with CTest:

$ RUNS=10 ctest -R luaL_gsub_test --verbose
<snipped>
1: Done 10 runs in 0 second(s)

By default the same number of runs is used:

$ ctest -R luaL_gsub_test --verbose
<snipped>
1: Done 5 runs in 0 second(s)

See commit that adds passing a number of test runs to CTest.

It is worth to say, that there is an issue in CMake, where people requests forwarding command-line arguments to test executables.

Upvotes: 0

Kevin
Kevin

Reputation: 1483

I've figured out a way to do it (using the Fundamental theorem of software engineering). It's not as simple as I'd like, but here it is.

First, create a file ${CMAKE_SOURCE_DIR}/cmake/RunTests.cmake with the content

if(NOT DEFINED ENV{TESTS_ARGUMENTS})
    set(ENV{TESTS_ARGUMENTS} "--default-arguments")
endif()
execute_process(COMMAND ${TEST_EXECUTABLE} $ENV{TESTS_ARGUMENTS} RESULT_VARIABLE result)
if(NOT "${result}" STREQUAL "0")
    message(FATAL_ERROR "Test failed with return value '${result}'")
endif()

Then, when you add the test, use

add_test(
    NAME MyTest
    COMMAND ${CMAKE_COMMAND} -DTEST_EXECUTABLE=$<TARGET_FILE:MyTest> -P ${CMAKE_SOURCE_DIR}/cmake/RunTests.cmake
)

Finally, you can run the test with custom arguments using

cmake -E env TESTS_ARGUMENTS="--custom-arguments" ctest

Note that if you use bash, you can simplify this to

TESTS_ARGUMENTS="--custom-arguments" ctest

There are some problems with this approach, e.g. it ignores the WILL_FAIL property of the tests. Of course I wish it could be as simple as calling ctest -- --custom-arguments, but, as the Stones said, You can't always get what you want.

Upvotes: 10

Bamboo
Bamboo

Reputation: 973

matthieu's answer gave me the clue to get it to work for me.

For my code I did the following:

Type the command ctest -V -R TestMembraneCellCrypt -N to get the output:

...
488: Test command: path/to/ctest/executable/TestMembraneCellCrypt
Labels: Continuous_project_ChasteMembrane
  Test #488: TestMembraneCellCrypt
...

Then I copied the Test command and provided the arguments there:

path/to/ctest/executable/TestMembraneCellCrypt -e 2 -em 5 -ct 10

I'll note that the package I'm using (Chaste), is pretty large so there might be things going on that I don't know about.

Upvotes: 0

matthieu
matthieu

Reputation: 1592

I'm not sure I fully understand what you want, but I still can give you a way to pass arguments to tests in CTest, at runtime.

I'll give you an example, with CTK (the Common Toolkit, https://github.com/commontk/CTK):

In the build dir (ex: CTK-build/CTK-build, it's a superbuild), if I run: ('-V' for Verbose, and '-N' for View Mode only)

ctest -R ctkVTKDataSetArrayComboBoxTest1 -V -N

I get:

UpdateCTestConfiguration  from : /CTK-build/CTK-build/DartConfiguration.tcl
Parse Config file:/CTK-build/CTK-build/DartConfiguration.tcl
 Add coverage exclude regular expressions.
 Add coverage exclude: /CMakeFiles/CMakeTmp/
 Add coverage exclude: .*/moc_.*
 Add coverage exclude: .*/ui_.*
 Add coverage exclude: .*/Testing/.*
 Add coverage exclude: .*/CMakeExternals/.*
 Add coverage exclude: ./ctkPixmapIconEngine.*
 Add coverage exclude: ./ctkIconEngine.*
UpdateCTestConfiguration  from :/CTK-build/CTK-build/DartConfiguration.tcl
Parse Config file:/CTK-build/CTK-build/DartConfiguration.tcl
Test project /CTK-build/CTK-build
Constructing a list of tests
Done constructing a list of tests

178: Test command: /CTK-build/CTK-build/bin/CTKVisualizationVTKWidgetsCppTests "ctkVTKDataSetArrayComboBoxTest1"
Labels: CTKVisualizationVTKWidgets
  Test #178: ctkVTKDataSetArrayComboBoxTest1

Total Tests: 1

You can copy-paste the "Test command" in your terminal:

/CTK-build/CTK-build/bin/CTKVisualizationVTKWidgetsCppTests "ctkVTKDataSetArrayComboBoxTest1"

And add the arguments, for example "-I" for interactive testing:

/CTK-build/CTK-build/bin/CTKVisualizationVTKWidgetsCppTests "ctkVTKDataSetArrayComboBoxTest1" "-I"

Tell me if it helps.

Upvotes: 0

Related Questions