Google Tests ValuesIn with a global vector

Question

I'm trying to run a bunch of google tests based off configuration files in a directory. This way I can just add a new file and run the tests without having to add it to my parametrized test and recompile. Here is my code:

typedef std::pair<QString, int> TestParam;

QString generatedLogic;

std::vector<TestParam> badExpressionTests;

class LogicBuilderTest : public ::testing::TestWithParam<TestParam> {};

GTEST_API_ int main(int argc, char **argv) {
    testing::InitGoogleTest(&argc, argv);

    ::generatedLogic = "/home/mitydsp/trunk/System/logicExecutionEngine/engine/include/GeneratedLogic.h";

    QString dir = "/home/mitydsp/trunk/System/logicExecutionEngine/unittest/expressions/bad/";
    QDirIterator it(dir, QStringList() << "*.txt", QDir::Files, QDirIterator::Subdirectories);

    while (it.hasNext()) {
        QString path = it.next();
        QStringList nameParts = it.fileName().split("_");
        int exitCode = nameParts[0].toInt();
        ::badExpressionTests.push_back(TestParam(path, exitCode));
    }

    std::cout << "Size of vector: " << badExpressionTests.size() << std::endl;

    return RUN_ALL_TESTS();
}

/**
 * Run parameterized test
 */
TEST_P(LogicBuilderTest, TestExitWithCode)
{    
    ::testing::FLAGS_gtest_death_test_style = "threadsafe";

    // Simulate fake main()
    char  arg0[] = "logicTest";
    char* argv[] = { &arg0[0], NULL };
    int argc = (int)(sizeof(argv) / sizeof(argv[0])) - 1;

    EXPECT_EXIT({
        // Need to run Qt Application because logic builder uses async QTimer::singleShot()
        QCoreApplication app(argc, argv);

        // Create a logic builder instance
        LogicBuilder logicBuilder(
            (int) ParameterStoreInterface::DEFAULT_PARAM_STORE_VALUES_KEY,
            (int) ParameterStoreInterface::DEFAULT_PARAM_STORE_NAMES_KEY,
            (int) ParameterStoreInterface::DEFAULT_PARAM_STORE_LOCK_KEY,
            QString(GetParam().first),
            QStringList(""),
            QStringList(generatedLogic),
            QString(LogicBuilder::DEFAULT_INTERMEDIATE_DIR),
            QString(LogicBuilder::DEFAULT_OUTPUT_SRC),
            QString(LogicBuilder::DEFAULT_OUTPUT_LIB),
            true );

        app.exec();
    }, ::testing::ExitedWithCode(GetParam().second), "");
} 

INSTANTIATE_TEST_CASE_P(TestBadExpressions, LogicBuilderTest,
                        ::testing::ValuesIn(::badExpressionTests));

When I run this, it shows that 0 tests are being ran even though the vector says its size is two. How come these parametrized tests are not being ran?

Size of vector: 2
[==========] Running 0 tests from 0 test cases.
[==========] 0 tests from 0 test cases ran. (0 ms total)
[  PASSED  ] 0 tests.

Originally I was running the tests by manually defining the configuration files but I don't like this:

    INSTANTIATE_TEST_CASE_P(TestBadExpressions, LogicBuilderTest,
                        ::testing::Values(
                            TestParam("unittest/expressions/bad/17_no_lhs.txt", LogicBuilder::LB_VARIABLE_NO_LHS),
                            TestParam("unittest/expressions/bad/25_incomplete_rhs.txt", LogicBuilder::LB_PARSE_ERROR)
                        ));

Answer 1

we spent some hours on this issue and thought this snippet will be helpful.

#include <iostream>
#include <string>
#include <vector>
using namespace std;
#include "gtest/gtest.h"

GTEST_API_ main( int argc, char* argv[] ) {
   testing::InitGoogleTest( &argc, argv );
   return RUN_ALL_TESTS();
}

struct test_parms_s {
   string         name;             // Simple name of this test
   string         description;      // Description of this test
   void*          function_parms;   // Pointer to test function specific data
};

std::vector<test_parms_s*> BuildTestData() {
   test_parms_s* pTestDataA = new test_parms_s(); 
   test_parms_s* pTestDataB = new test_parms_s(); 
   test_parms_s* pTestDataC = new test_parms_s();

   pTestDataA->name.assign("testA");
   pTestDataA->description.assign("testA_desc");
   pTestDataA->function_parms = NULL;

   pTestDataB->name.assign("testB");
   pTestDataB->description.assign("testB_desc");
   pTestDataB->function_parms = NULL;

   pTestDataC->name.assign("testC");
   pTestDataC->description.assign("testC_desc");
   pTestDataC->function_parms = NULL;

   std::vector<test_parms_s*> values;
   values.push_back(pTestDataA);
   values.push_back(pTestDataB);
   values.push_back(pTestDataC);
   cout << "BuildTestData" << endl;

   return values;
}
//------------------------------------------------------------------------
class Testy {
  private:
     string       testname_;
  public:
     Testy( string testname );
     ~Testy();
     string   GetTestName();
     void     SetTestName( string testname );
};
Testy::Testy( string testname ) {
   testname_.assign(testname);
}
Testy::~Testy() {}
string Testy::GetTestName() { return testname_; }
void Testy::SetTestName( string testname ) { testname_ = testname; }
//------------------------------------------------
class TestFixture : public ::testing::TestWithParam<test_parms_s*> {
   protected:
     Testy* testy;      
     virtual void SetUp() {
        testy = new Testy( "Test");
     }
     virtual void TearDown() {
        delete( testy );
     }
};
//------------------------------------------------
TEST_P( TestFixture, ParamTest ) {
   test_parms_s* pTestParms = GetParam();
   cout << pTestParms->name << endl;
} 
//------------------------------------------------
INSTANTIATE_TEST_CASE_P( TestFixture_instance, TestFixture,                     
                         ::testing::ValuesIn(BuildTestData()));
//========================================================================

the output will be

BuildTestData
[==========] Running 3 tests from 1 test case.
[----------] Global test environment set-up.
[----------] 3 tests from TestFixture_instance/TestFixture
[ RUN      ] TestFixture_instance/TestFixture.ParamTest/0
testA
[       OK ] TestFixture_instance/TestFixture.ParamTest/0 (0 ms)
[ RUN      ] TestFixture_instance/TestFixture.ParamTest/1
testB
[       OK ] TestFixture_instance/TestFixture.ParamTest/1 (0 ms)
[ RUN      ] TestFixture_instance/TestFixture.ParamTest/2
testC
[       OK ] TestFixture_instance/TestFixture.ParamTest/2 (0 ms)
[----------] 3 tests from TestFixture_instance/TestFixture (0 ms total)

[----------] Global test environment tear-down
[==========] 3 tests from 1 test case ran. (0 ms total)
[  PASSED  ] 3 tests.

Answer 2

I've spent the last hour trying to figure this out and then as soon as I post I come up with the solution. I'll leave this here because it may help someone in the future:

Instead of trying to create the list of files in main, pass the ValuesIn a function which loads the files and returns a vector.

std::vector<TestParam> GetFilesInDir() 
{
    std::vector<TestParam> values;
    QString dir = "/home/mitydsp/trunk/System/logicExecutionEngine/unittest/expressions/bad/";
    QDirIterator it(dir, QStringList() << "*.txt", QDir::Files, QDirIterator::Subdirectories);

    while (it.hasNext()) {
        QString path = it.next();
        QStringList nameParts = it.fileName().split("_");
        int exitCode = nameParts[0].toInt();
        values.push_back(TestParam(path, exitCode));
    }

    return values;
}

INSTANTIATE_TEST_CASE_P(TestBadExpressions, LogicBuilderTest,
                        ::testing::ValuesIn(GetFilesInDir()));

Thanks to Maksim Solovjov for reminding me that INSTANTIATE_TEST_CASE_P macro was being executed before main.

Answer 3

I'm not an expert in manually instantiating Google Tests, but the way you did it can't possibly work: you call INSTANTIATE_TEST_CASE_P in the static context, so it is being evaluated before main is ever called.

You could try moving instantiation just before RUN_ALL_TESTS; however, I don't know what that macro does and this might be illegal. In that case, I think that you can't create your tests with INSTANTIATE_TEST_CASE_P in the dynamic way that you want.