CODEWITHSUNDEEP
c#.netprotobuf-net
damageboy
damageboy

Reputation: 2117

How to minimize Protobuf.NET memory consumption

I'm serializing large quantities of classes, most of them as references (as I have multiple refernces to the same classes in my data-structures) using Protobuf.NET.

All of the serialized classes are serialized by using ImplicitFirends.AllFields to make sure everything is dumped.

In my current test case, I have 53 files with a total sum of 500MB.

When I read this data using a Protobuf.NET de-serializer my private bytes / resident memory shoots up to 9GB and STAYS there (i.e. this is not temporary memory released/GC'd after de-serialization).

Another peculiar thing is that if I re-write (re-serialize) all of the data, it still stays at the same size.

Does this x20 blow up in memory make sense?

Upvotes: 4

Views: 1323

Answers (1)

Marc Gravell
Marc Gravell

Reputation: 1063338

I'll run some tests, but it sounds like it is the buffer-pool that it keeps around to avoid allocations. In addition to testing, I'll add a "dump the pool" method in the next release (and disable?). In fact, thinking about it: I'll change these to WeakReference, so they can be re-used while they exist, but still collected.

You might also find that (as an option on the contract) using "group" encoding for sub-objects and lists makes a big reduction here; length-prefixed is the default (and Google's preferred option), but groups are much more efficient to write, as they can be written direct without any buffering. Let me know if you would like an example of this. Protobuf-net is designed such that switching between them is not a breaking change, but other protobuf clients won't be as forgiving, so trickier if ou are using protobuf for interop to other systems.

Upvotes: 5

Related Questions