CODEWITHSUNDEEP
c++qtserializationeigen
Rao
Rao

Reputation: 61

Eigen - How to create the smallest serialized MatrixXf

I am serializing relatively large MatrixXf to file via similar code to this -- https://github.com/github188/vidy_old/blob/01c0aa4242299f7f1056edf0aa60b92177dfcfe6/3rdparty/openbr/core/eigenutils.h

When I do this using Qt, it creates a 3-30MB files depending on the matrix.

If saving space were my #1 objective, what are some things I could consider doing?

  1. Is there another form of serialization that I would expect to see smaller file size? (I might try boost to see how that differs - https://gist.github.com/ShigekiKarita/4db2b0ffb207322c1324)
  2. It seems that the serialized matrix could be compressed a bit if I could live with having less decimal places of floating point precision so it doesn't have to store as much on disk. Does anyone know of such a technique?

Thanks you,

Rao

Upvotes: 0

Views: 187

Answers (2)

Mr. White
Mr. White

Reputation: 601

Boost serialization, together with compression, using the boost's iostream filtering should give you a flexible solution, with small file sizes.

The binary_archive of Boost serialization is quite compact, and allows file-versioning, which can be handy.

Upvotes: 1

Lanting
Lanting

Reputation: 3068

Independent of the serialization you're using, boost provides filtering streams, that work exactly like regular istreams and ostreams, however add compression on the fly. You could simply replace the streams in your example with the filtered ones, and should have working compression.

http://www.boost.org/doc/libs/1_64_0/libs/iostreams/doc/classes/gzip.html

#include <fstream>
#include <iostream>
#include <boost/iostreams/filtering_streambuf.hpp>
#include <boost/iostreams/copy.hpp>
#include <boost/iostreams/filter/gzip.hpp>

int main() 
{
    using namespace std;

    ifstream file("hello.gz", ios_base::in | ios_base::binary);
    filtering_streambuf<input> in;
    in.push(gzip_decompressor());
    in.push(file);
    boost::iostreams::copy(in, cout);
}

Upvotes: 0

Related Questions