(De/)Serialization as an interface for file-based data exchange in C++

Question

I'm trying to find some general understanding of proper use of serialization for data transfer. Imagine we have two binaries. Each binary has some internal representation of data which fits its internal use model and the representation differs across binaries. Now we want to pass some data from one binary to other.

Two options:

• Use an auxiliary class which have serilalization/deserialization methods. The class is optimized for storage and holds all the file representation logic in one place.
• Use custom reading/writing in a standardized format from the internal structures in each binary separately. This allows to avoid overhead for creation of an auxiliary class instance and double copying.

Q:

• What speculations I need to make in order to understand applicability of the two approaches in a particular case?
• Is there a pattern which mixes the solutions in a way which preserves pros of both of them?

Answer 1

The practical reality is that there is very little to differentiate the two approaches that you have suggested. But there are a number of important implementation trade-offs that you need to consider. Including:

1. Should the serialization format be human readable or binary. Human readable might by CSV, XML, JSON, YAML. Binary could be ASN.1, custom format, output from serialization library.
2. How extensible should the format be with respect to future changes. Do you need a file format version number? Should you allow for arbitrary fields?
3. Does it need to be standards based and will other applications in future also participate in this information interchange.
4. How secure does the interchange need to be. Should it use SSL/TLS, should the data be encrypted etc.
5. How reliable and/or performant dose the interchange need to be. XML comes with XSD for validating the data but is not nearly as fast to parse as other formats. If you need reliable delivery should you use a message queue (ActiveMQ, MSMQ etc)?
6. How are the binaries going to share the data: watched folder, Memory mapped file, named pipe, sockets, message queue, boost::asio, Thrift, XML RPC, HTTP, SOAP, REST etc

Whether you serialize a class using say boost::serialization or google protocol buffers or whether you say write manual XML DOM code to read and write the data is a smaller detail of the more important considerations in your design. However, in my experience, if you have the flexibility to use a custom data format using serialization code is more maintainable, performs better and has fewer quality issues than home grown solutions.