How can I switch element and container types for purpose of benchmarking, without producing a large number of test client applications?

Question

I have a ClientInterface class, that uses the Strategy pattern to organize two complex algorithms conforming to interfaces Abase and Bbase, respectively. The ClientInterface agglomerates (via composition) the data on which the algorithms operate, which needs to conform to the Data interface.

What I tried to do is to have a single ClientInterface class, which is able to choose different Strategies and Data implementations at run-time. The algorithms and data implementations are chosen using the Factory Method which reads the strings from an input file and selects the algorithm and data implementation in the ClientInterface constructor. The run-time choice of data and algorithms is not provided in the code model below.

The Data implementation can be based on a map, a list, an unordered_map , etc. to test how does the efficiency of two complex algorithms (Abase and Bbase implemented Strategies) change with different containers used for the Data.

Additionally, the Data agglomerates different Elements (ElementBase implementations). Different element implementations will also have significant impact on the efficiency of theh ClientInterface, but the Elements are really disjoint types with implementations coming from different libraries. I know this for a fact, since profiling the existing application shows the Element operation to be one of the bottlenecks.

I know that if I use polymorphism with containers, there is "boost/ptr_container" out there, but the Data will store hundreds of thousands, if not millions of Elements. Using polymorphism for Elements in this case will have a significant overhead on the ClientInterface, but if I choose to make data a class Template for the Element type, I will end up statically defining the ClientInterface class, which means producing a client application per each Element type at least.

Can I assume that for the same number of Elements and the ClientInterface configuration obtained at run-time, the overhead induced by the use of polymorphism for the Element type will have the same impact on all configurations of the Data and Algorithm implementations? In this case, I can run the automated tests, decide on the configuration of the Data implementation and the Element implementation, and define a statically configured EfficientClientInterface to be used in the productive code?

Goal: I have a test harness prepared, and what I am trying to do is to automatize the testing on the family of test cases, since changing the Algorithms and Elements at run-time, allows me to use a single application in a loop, which is configured at run-time and whose output is measured for efficiency. In the real implementation, I am dealing with at least 6 algorithm interfaces, 3-4 Data implementations, and I estimate 3 Element implementations at least.

So, my questions are:

1) How can an Element support different operations when overloading is not working for return types? If I make the operation a template, it needs to be defined at compile-time, which messes with my automated testing procedure.

2) How can I design this code better to achieve the goal?

3) Is there a better overall approach to this problem?

Here is the code model:

#include <iostream>
#include <memory>

class ElementOpResultFirst
{};

class ElementOpResultSecond
{};

class ElementBase
{
    public: 

        // Overloading does not allow different representation of the solution for the element operation.
        virtual ElementOpResultFirst elementOperation() = 0; 
        //virtual ElementOpResultSecond elementOperation() = 0; 

}; 

class InterestingElement
: 
    public ElementBase
{
    public: 
        ElementOpResultFirst elementOperation() 
        {
            // Implementation dependant operation code. 

            return ElementOpResultFirst(); 
        } 
        //ElementOpResultSecond elementOperation() 
        //{
            //// Implementation dependant operation code.

            //return ElementOpResultSecond(); 
        //} 
};

class EfficientElement
: 
    public ElementBase
{
    public: 
        ElementOpResultFirst elementOperation() 
        {
            // Implementation dependant operation code.

            return ElementOpResultFirst(); 
        } 
        //ElementOpResultSecond elementOperation() 
        //{
            //// Implementation dependant operation code.

            //return ElementOpResultSecond(); 
        //} 
};

class Data
{
    public: 
        virtual void insertElement(const ElementBase&) = 0;  
        virtual const ElementBase& getElement(int key) = 0;
};

class DataConcreteMap
:
    public Data
{
    // Map implementation

    public: 
        void insertElement(const ElementBase&)
        {
            // Insert element into the Map implementation.
        } 
        const ElementBase& getElement(int key)
        {
            // Get element from the Map implementation.
        } 
};

class DataConcreteVector
:
    public Data
{
    // Vector implementation

    public: 
        void insertElement(const ElementBase&)
        {
            // Insert element into the vector implementation.
        } 
        const ElementBase& getElement(int key)
        {
            // Get element from the Vector implementation
        } 
};

class Abase
{
    public: 
        virtual void aFunction() = 0; 
};

class Aconcrete
:
    public Abase
{
    public: 
        virtual void aFunction() 
        {
            std::cout << "Aconcrete::function() " << std::endl;
        }
};

class Bbase
{
    public: 
        virtual void bFunction(Data& data) = 0; 
};

class Bconcrete
:
    public Bbase
{
    public: 
        virtual void bFunction(Data& data) 
        {
            data.getElement(0); 
            std::cout << "Bconcrete::function() " << std::endl;
        }
};

// Add a static abstract factory for algorithm and data generation. 

class ClientInterface
{
    std::unique_ptr<Data>  data_; 
    std::unique_ptr<Abase> algorithmA_; 
    std::unique_ptr<Bbase>  algorithmB_; 

    public: 

        ClientInterface()
            :
                // A Factory Method is defined for Data, Abase and Bbase that 
                // produces the concrete type based on an entry in a text-file.
                data_ (std::unique_ptr<Data> (new DataConcreteMap())), 
                algorithmA_(std::unique_ptr<Abase> (new Aconcrete())),
                algorithmB_(std::unique_ptr<Bbase> (new Bconcrete()))
        {}

        void aFunction()
        {
            return algorithmA_->aFunction(); 
        }

        void bFunction()
        {
            return algorithmB_->bFunction(*data_); 
        }
};

// Single client code: both for testing and final version.
int main()
{
    ClientInterface cli; 

    cli.aFunction(); 
    cli.bFunction(); 

    return 0; 
};

Answer 1

What I tried to do is to have a single ClientInterface class, which is able to choose different Strategies and Data implementations at run-time. The algorithms and data implementations are chosen using the Factory Method which reads the strings from an input file and selects the algorithm and data implementation in the ClientInterface constructor. The run-time choice of data and algorithms is not provided in the code model below.

Sounds like you have the basis for some of it here: Either just produce a set of files to test with that produce the right different sets of inputs. Or refactor the Factory function so that the reading of the file and the strings are separate, so you can call your factory function [internals] with a a string from the code.

Can I assume that for the same number of Elements and the ClientInterface configuration obtained at run-time, the overhead induced by the use of polymorphism for the Element type will have the same impact on all configurations of the Data and Algorithm implementations? In this case, I can run the automated tests, decide on the configuration of the Data implementation and the Element implementation, and define a statically configured EfficientClientInterface to be used in the productive code?

I don't think you can make that assumption. Different implementations may well have different effects on the algorithms - copying a 100 byte string is significantly harder than copying a 4 byte integer, for example. So what the data is, and how it's organized will have some effect on the work you do. Of course, since you haven't described in much detail what your Elements actually contain, it's all guesswork.

1) How can an Element support different operations when overloading is not working for return types? If I make the operation a template, it needs to be defined at compile-time, which messes with my automated testing procedure.

Make a factory class that returns an ElementBase reference or pointer? That's my immediate reaction to this question, but again, the detail in your question is sufficiently vague that it's hard to say for sure.

In the real application, how does this work? Is it done by templates, then you'd better implement the testcode by templates, and fill it out with a selection of realistic variations on what you think the real system is likely to do.

2) How can I design this code better to achieve the goal?

Try to reuse the production code?

3) Is there a better overall approach to this problem?

Not sure yet.