Data Structure to hold multiple data types C++

Question

In my application there is a class (i.e. ItemData Class) which has more than 30 member variables of different types such as

int a;
int b;
std::string c;
float d;
double e;
double f;
char * g;

and many more.

My application needs to create large number of itemdata classes so that memory usage of the application is high. One special fact about this class is most of the instances of ItemData class has values for few member variables only.

Ex: Instance one may have values for a and b only. Instance two can have values for b,c and g only.

So to reduce the memory usage of my application, I need way to allocate memory only for member variables which has data when the instance is creating. So I though of having generic data structure which can access elements via position and store data on that. So ItemData class has something like below and store dynamically allocated data on that.(I have to maintain the position and information hold on each position seperately)

std::vector<void*> vec_DataArray;

If ItemData instance mItemData1 has values for a and d:

mItemData1.vec_DataArray[0] = new int(iValue);
mItemData1.vec_DataArray[3] = new float(fValue);

Can someone let me know whether this is a good approach to reduce the memory usage in my application? Is there any generic container which can hold multiple data types (for vec_DataArray) so that I can avoid void* to datatype conversion when accessing data.

Answer 1

C++ has many types of multiple data storage structure like mentioned above boost::any or boost::variant or the build it variant and any from std.

However, if you wanted to implement one for yourself. This class below I built can demonstrate something that you can build to store multiple types of data. If you remove the union class. You can store and get multiple types of data from just one variable. However, if you remove the union class. The memory usage will not be that efficient. A Union type could have solved many issues. However, if you wanted to store an std::string, into a union, you can't do that.

You can expand the type as needed.

If you use this you can do something like this: _cDynamic a; a = (int)124.

Then after a while, you don't know what was stored in the variable then you can do a.check(). check() will return the latest data type that was stored into the instance.

You can use getS(), getI(), getUI(), etc.. to get back the data type if you know for sure what was stored in there. There is no error checking on that. Or you can use: something = get(&variable). Or you can use getVoid() and a void pointer to the latest data type will be returned. If you use getVoid() you need to know the data type and cast it properly.

So if you stored an int into a as the latest value then doing a.check() will return an std::string of "int". For efficiency, every time you set a new data type to a, a does not delete what was store last. It does not need to anyway for anything that is defined in the union. Nonetheless, for the string class, it would be different. However, in your case, you needed memory efficiency. Thus, you can just mod the code so that if anything that is not an std::string is being set to instance then you just reset the vString variable to empty. Which should help to free up some of the needed memory. If you don't utilize string just remove the string type from the class. That will help reduce memory usage as the union type will only use up so much memory that would be as big as the biggest data type that it was defined to hold. Also, you can remove any unnecessary function that you don't use from the class.

Also, current can be an unsigned char to use even less memory. You can just number type your system. However, I used std::string so that it can easily be referenced by looking at it.

#include <map>
#include <iostream>

union _uV {
    int           i;     
    unsigned int ui;     
    long          l;     
    double        d;     
    float         f;     
    char          c;     
    bool          b;     
};

class _cDynamic{ 
    std::string vString;     
    std::string current = "null";

    static std::string intname;
    static std::string uintname;
    static std::string longname;
    static std::string doublename;
    static std::string floatname;
    static std::string charname;
    static std::string boolname;
    static std::string stringname;

    _uV vNorm;

    template<class T>
    void set(const T &v, const std::string vname){
        const void * p = &v;

        if ( vname == intname ) {
            vNorm.i = *static_cast<const int *> (p);
            current = "int";
        } else if ( vname == uintname ){
            vNorm.ui = *static_cast<const unsigned int *> (p);
            current = "uint";
        } else if ( vname == longname ){
            vNorm.l = *static_cast<const long *> (p);
            current = "long";
        } else if ( vname == doublename ){
            vNorm.d = *static_cast<const double *> (p);
            current = "double";
        } else if ( vname == floatname ){
            vNorm.f = *static_cast<const float *> (p);
            current = "float";
        } else if ( vname == charname ){
            vNorm.c = *static_cast<const char *> (p);
            current = "char";
        } else if ( vname == boolname ){
            vNorm.b = *static_cast<const bool *> (p);
            current = "bool";
        } else if ( vname == stringname ){
            vString = *static_cast<const std::string *> (p);
            current = "string";
        } else {
            throw "unsupported type.";
        }
    };

    public:

    template<class T>
    _cDynamic operator=(const T &v)
    {    
        std::string thisname = typeid(this).name();
        std::string vname = typeid(v).name();
        if ( vname == thisname ) return *this;
        set(v, vname);
        return *this;
    };

    _cDynamic(){};
    template<class T>
    _cDynamic(const T &v)
    {
        std::string vname = typeid(v).name();
        set(v, vname);
    };

    bool get(int &v){
        if ( current == "int" ){
            v = vNorm.i;
            return true;
        } else return false;
    };

    bool get(unsigned int &v){
        if ( current == "uint" ){
            v = vNorm.ui;
            return true;
        } else return false;
    };    

    bool get(long &v){
        if ( current == "long" ){
            v = vNorm.l;
            return true;
        } else return false;
    };

    bool get(double &v){
        if ( current == "double" ){
            v = vNorm.d;
            return true;
        } else return false;
    };

    bool get(float &v){
        if ( current == "float" ){
            v = vNorm.f;
            return true;
        } else return false;
    };

    bool get(char &v){
        if ( current == "char" ){
            v = vNorm.c;
            return true;
        } else return false;
    };


    bool get(bool &v){
        if ( current == "bool" ){
            v = vNorm.b;
            return true;
        } else return false;
    };

    bool get(std::string &v){
        if ( current == "string" ){
            v = vString;
            return true;
        } else return false;
    };

    void * getVoid(){
        void * p;
        if ( current == "int" ){ 
            p = &vNorm.i;
            return p;
        } else if ( current == "uint" ){
            p = &vNorm.ui;
            return p;
        } else if ( current == "long" ){
            p = &vNorm.l;
             return p;
        } else if ( current == "double" ){
            p = &vNorm.d;
              return p;
         } else if ( current == "float" ){
            p = &vNorm.f;
              return p;
        } else if ( current == "char" ){
            p = &vNorm.c;
            return p;
        } else if ( current == "bool" ){
            p = &vNorm.b;
            return p;
        } else if ( current == "string" ){
            p = &vString;
            return p;
        } else throw "Unitialized";
    };

    int getI(){ return vNorm.i; };
    unsigned int getUI() { return vNorm.ui; };
    long getL() { return vNorm.l; };
    double getD() { return vNorm.d; };
    float getF() { return vNorm.f; };
    char getC() { return vNorm.c; };
    bool getB() { return vNorm.b; };
    std::string getS() { return vString; };


    std::string check() { return current; };
};

std::string _cDynamic::intname = typeid(int).name();
std::string _cDynamic::uintname = typeid(unsigned int).name();
std::string _cDynamic::longname = typeid(long).name();
std::string _cDynamic::doublename = typeid(double).name();
std::string _cDynamic::floatname = typeid(float).name();
std::string _cDynamic::charname = typeid(char).name();
std::string _cDynamic::boolname = typeid(bool).name();
std::string _cDynamic::stringname = typeid(std::string).name();

int main(){

    int test; bool ok;
    std::map <int, _cDynamic> a = { {0, std::string("kevin")} };
    std::cout << a[0].check() << " " << a[0].getS() << std::endl;
    a[0] = (int) 1234;
    std::cout << a[0].check() << " " << a[0].getI() << std::endl;
    a[0] = std::string("This is a string");
    std::cout << a[0].check() << " " << a[0].getS() << std::endl;
    a[0] = (float)124.55;
    std::cout << a[0].check() << " " << a[0].getF() << std::endl;
    a[0] = (bool)true;
    std::cout << a[0].check() << " " << a[0].getB() << std::endl;
    a[0] = 'z';
    std::cout << a[0].check() << " " << a[0].getC() << std::endl;


    a[0] = (int)1234;
    ok = a[0].get(test);
    std::cout << a[0].check() << ": is ok("<< ok << ") " << test << std::endl;

    a[0] = (int)2345;
    test = *static_cast<int*>(a[0].getVoid());
    std::cout << a[0].check() << " " << test << std::endl;

    return 0;
}

For another example below. You can store many types into one variable. This class below is almost similar to the one above. However, each time you assign something to it, it will store it and flag that it has that type of data. This may not fit your case, however, if you mod the code below properly, you may be able to combo the data in such a way that you may save memory instead of utilizing more. You can also mod the code below to be able to store more than one string if needed or more than one of any other data type.

For the code below, you can use variable.stored() to get a map of what stored. The map will contain 8 indexes correspond to what data was stored into the instance. You can use variable.latest() to see what data storage last. However, unlike current from above, the latest does not have much effect on this one.

Similar to the above, you can use get(&v) or getI(), getUI(), and etc.. to get the data out. There is no reason for the getVoid() for this one. There also isn't an error checking for anything that is not get();

Note: if you already store a data type into the instance and then you later store the same type into it, the new data will replace the old data.

An example case for the one below is:

_cDynamicx a;
a = (int)123;
a = std::string("me");

if ( a.stored()["string"] == true ){
    std::cout << a.getS() << std::endl;
} else {
    std::cout << "A did not have string.";
}

Both the functions are built by me and are in my library which I licensed them as MIT so feel free to mod and use them if you like them.

class _cDynamicx{ 
    int vInt;
    unsigned int vUInt;
    long vLong;
    double vDouble;
    float vFloat;
    char vChar;
    bool vBool;
    std::string vString;

    std::string _latest = "null";
    std::map <std::string, bool> _stored = {
        {"int", false},
        {"uint", false},
        {"long", false},
        {"double", false},
        {"float", false},
        {"char", false},
        {"bool", false},
        {"string", false},    
    };

    static std::string intname;
    static std::string uintname;
    static std::string longname;
    static std::string doublename;
    static std::string floatname;
    static std::string charname;
    static std::string boolname;
    static std::string stringname;


    template<class T>
    void set(const T &v, const std::string &vname){
        const void * p = &v;

        if ( vname == intname ) {
            vInt = *static_cast<const int *> (p);
            _latest = "int";
        } else if ( vname == uintname ){
            vUInt = *static_cast<const unsigned int *> (p);
            _latest = "uint";
        } else if ( vname == longname ){
            vLong = *static_cast<const long *> (p);
            _latest = "long";
        } else if ( vname == doublename ){
            vDouble = *static_cast<const double *> (p);
            _latest = "double";
        } else if ( vname == floatname ){
            vFloat = *static_cast<const float *> (p);
            _latest = "float";
        } else if ( vname == charname ){
            vChar = *static_cast<const char *> (p);
            _latest = "char";
        } else if ( vname == boolname ){
            vBool = *static_cast<const bool *> (p);
            _latest = "bool";
        } else if ( vname == stringname ){
            vString = *static_cast<const std::string *> (p);
            _latest = "string";
        } else {
            throw "unsupported type.";
        }

        _stored[_latest] = true;
    };

    public:

    template<class T>
    _cDynamicx operator=(const T &v)
    {    
        std::string thisname = typeid(this).name();
        std::string vname = typeid(v).name();
        if ( vname == thisname ) return *this;
        set(v, vname);
        return *this;
    };

    _cDynamicx(){};
    template<class T>
    _cDynamicx(const T &v){
        std::string vname = typeid(v).name();
        set(v, vname);
    };

    bool get(int &v){
        if ( _stored["int"] == true ){
            v = vInt;
            return true;
        } else return false;
    };

    bool get(unsigned int &v){
        if ( _stored["uint"] == true ){
            v = vUInt;
            return true;
        } else return false;
    };    

    bool get(long &v){
        if ( _stored["long"] == true ){
            v = vLong;
            return true;
        } else return false;
    };

    bool get(double &v){
        if ( _stored["double"] == true ){
            v = vDouble;
            return true;
        } else return false;
    };

    bool get(float &v){
        if ( _stored["float"] == true ){
            v = vFloat;
            return true;
        } else return false;
    };

    bool get(char &v){
        if ( _stored["char"] == true ){
            v = vChar;
            return true;
        } else return false;
    };


    bool get(bool &v){
        if ( _stored["bool"] == true ){
            v = vBool;
            return true;
        } else return false;
    };

    bool get(std::string &v){
        if ( _stored["string"] == true ){
            v = vString;
            return true;
        } else return false;
    };

    int getI(){ return vInt; };
    unsigned int getUI() { return vUInt; };
    long getL() { return vLong; };
    double getD() { return vDouble; };
    float getF() { return vFloat; };
    char getC() { return vChar; };
    bool getB() { return vBool; };
    std::string getS() { return vString; };


    std::string latest() { return _latest; };
    std::map<std::string, bool> stored() { return _stored; };

};

std::string _cDynamicx::intname = typeid(int).name();
std::string _cDynamicx::uintname = typeid(unsigned int).name();
std::string _cDynamicx::longname = typeid(long).name();
std::string _cDynamicx::doublename = typeid(double).name();
std::string _cDynamicx::floatname = typeid(float).name();
std::string _cDynamicx::charname = typeid(char).name();
std::string _cDynamicx::boolname = typeid(bool).name();
std::string _cDynamicx::stringname = typeid(std::string).name();

Answer 2

I think you should avoid using a data-structure that can hold multiple types of data if you can and consider a different design.

For example you could consider using something a bit like the Flyweight pattern and pull the "extrinsic" state out of the ItemData class leaving only the "intrinsic" state. It feels a bit non object-oriented to me but it might meet your needs. For example you could keep separate maps from item index to data. Whether this will help with your memory usage depends on how sparse your data is.

#include <unordered_map>
#include <iostream>

class Item {
 private:
  std::string type_;  // "intrinsic" state
 public:
  Item(const std::string &type) : type_(type) {} 

  // pass in "extrinsic" state
  void someOperation(std::string color, double speed) {

    // do something using both "intrinsic" and "extrinsic" state...
    std::cout << color << " " << type_ << " moving at " << speed << "mph\n";
  }
};

class AnimalSim {
 private:
  std::unordered_map<int, std::string> duck_colors_;   // Store extrinsic
  std::unordered_map<int, std::string> sheep_colors_;  // state separately
  std::unordered_map<int, double>      duck_speeds_;   // in a more
  std::unordered_map<int, double>      sheep_speeds_;  // efficent way.
 public:
  void run();
  std::string getDuckColor(int duck_index) const;
  std::string getSheepColor(int sheep_index) const;
  double      getDuckSpeed(int duck_index) const;
  double      getSheepSpeed(int sheep_index) const;
};

void
AnimalSim::run() {
  auto duck = Item{"duck"};    // Create `Flyweight` objects that can be shared.
  auto sheep = Item{"sheep"};  // Should probably be done by a factory with a cache.

  // Create duck 0
  duck_colors_.emplace(0, "red");
  duck_speeds_.emplace(0, 150.0);

  // Create duck 1 - has no speed
  duck_colors_.emplace(1, "green");
  size_t num_ducks = 2;

  // Create sheep 0 - has no color
  sheep_speeds_.emplace(0, 100.0);
  size_t num_sheep = 1;

  // Do something with all the ducks
  for(size_t i = 0; i != num_ducks; ++i)
    duck.someOperation(getDuckColor(i), getDuckSpeed(i));

  // Do something with all the sheep
  for(size_t i = 0; i != num_sheep; ++i)
    sheep.someOperation(getSheepColor(i), getSheepSpeed(i));    
}

std::string
AnimalSim::getDuckColor(int duck_index) const {
  auto color_itr = duck_colors_.find(duck_index);
  return color_itr != duck_colors_.end() ? color_itr->second : "black"; 
}

std::string
AnimalSim::getSheepColor(int sheep_index) const {
  auto color_itr = sheep_colors_.find(sheep_index);
  return color_itr != sheep_colors_.end() ? color_itr->second : "white"; 
}

double
AnimalSim::getDuckSpeed(int duck_index) const {
  auto speed_itr = duck_speeds_.find(duck_index);
  return speed_itr != duck_speeds_.end() ? speed_itr->second : 0.0; 
}

double
AnimalSim::getSheepSpeed(int sheep_index) const {
  auto speed_itr = sheep_speeds_.find(sheep_index);
  return speed_itr != sheep_speeds_.end() ? speed_itr->second : 0.0; 
}

int main() {
  AnimalSim animal_sim;
  animal_sim.run();
}

Live demo.

Edit: I see you are loading data from a database. In which case I wonder why you are not just loading-on-demand from the database when you need to?

Answer 3

You have a company ID and a set of company descriptors of various types, wherein each kind of descriptor has a unique tag (the "column"), and some kind of mapping between the ID and the set. So you have something like map<company_id_t, set<company_detail_t>>. Note the set doesn't have to be std::set, you can easily build your own e.g. vector-based solution, but conceptually it is a set.

company_detail_t could be a pair of detail_tag_t and detail_value_t. The set is keyed on the tag only, so no two details have the same tag. The value could be

1. a union
2. a boost::variant
3. a boost_any
4. a pointer to a detail_value_base class object that has different subclasses. In this case detail_tag_t could actually be a result of virtual detail_tag_t detail_value_base::getTag() function.

Answer 4

One possible solution to your problem is to defer the decision about the actual composition of the record until you need to process it. For instance, if you are parsing a text string fron the file, do not parse it while reading; pass around or store the actual text line instead. Parse it on-demand, when you need the process the fields. This is known as "lazy parsing", and it avoids the problem of storing indeterminate type by...not determining the type.

Answer 5

Allocating primitive types on the heap is a very, very bad approach. You will get huge memory management overhead, added indirection, and your memory usage will go through the roof because allocation of each primitive type will take at least 3 addresses in memory pool. Use (discriminated) unions for primitive types. If you create a union with constructed types, you have to manage calls to constructors and destructors manually.