CODEWITHSUNDEEP
serializationprotocol-buffersflatbuffers
Florian
Florian

Reputation: 51

How to use zero-copy serialization libraries without moving the data?

I am looking for a binary serialization library because JSON turned out to be too slow for our use case. Some lib devs (Flatbuffers, Cap'n Proto, ..) argue against Protobuf because it doesn't follow the zero-copy idea (e.g. 1 and 2). When I think about how to integrate any of those libs into our code, I see a problem with the "Usable as mutable state" point in 1. This basically says that the zero-copy feature leads to (mostly) immutable objects. So I need a copy of all (or most of) the fields in my own class to be able to modify/write the data (the "mutable state"). In this case I will always need to copy my internal state to the serializer object - which does not feel like zero-copy.

How is that different for Protobuf?* I could use the object as my mutable internal state (or is there anything wrong with that?) and from there it also takes me just one copy to get the serialized object.

* for any use case where you not only read the data..

Upvotes: 4

Views: 4444

Answers (1)

Aardappel
Aardappel

Reputation: 6074

If your use case involves mutation, then indeed the advantage of zero-copy libraries shrinks compared to Protobuf.

FlatBuffers has 3 ways to do mutation:

  • If the mutation is only of scalars, these can be mutated in-place (--gen-mutable), which is very limiting but also very efficient.
  • If mutation is very infrequent, you can still do them in-place using reflection. This allows more kinds of mutations but is slow and complex to use.
  • You can use the object API (--gen-object-api). This generates a similar API to that of Protobuf, which is fully mutable and easy to use, but of course does more memory allocation. There are still advantages of using this over Protobuf, as the API is more idiomatic C++, de/encoding is likely still faster, and if you have some part of your system that does not need to mutate, they can use the faster underlying API.

Upvotes: 2

Related Questions