CODEWITHSUNDEEP
c#.netmemoryprotocol-buffersprotobuf-net
browe
browe

Reputation: 198

Deserializing packed bytes to a Stream member with protobuf-net

Is there a way to deserialize a bytes field to a Stream member, without protobuf-net allocating a new (and potentially big) byte[] upfront?

I'm looking for something like this:

[ProtoContract]
public class Message
{
    [ProtoMember(1)]
    Stream Payload { get; set; }
}

Where the Stream could be backed by a pre-allocated buffer pool e.g. Microsoft.IO.RecyclableMemoryStream. Even after dropping down to ProtoReader for deserialization all I see is AppendBytes, which always allocates a buffer of field length. One has to drop even further to DirectReadBytes, which only operates directly on the message stream -- I'd like to avoid that.

As background, I'm using protobuf-net to serialize/deserialize messages across the wire. This is a middle-layer component for passing messages between clients, so the messages are really an envelope for an enclosed binary payload:

message Envelope {
  required string messageId = 1;
  map<string, string> headers = 2;
  bytes payload = 3;
}

The size of payload is restricted to ~2 MB, but large enough for the byte[] to land in the LOH.

Using a surrogate as in Protobuf-net: Serializing a 3rd party class with a Stream data member doesn't work because it simply wraps the same monolithic array.

One technique that should work is mentioned in Memory usage serializing chunked byte arrays with Protobuf-net, changing bytes to repeated bytes and relying on the sender to limit each chunk. This solution may be good enough, it'll prevent LOH allocation, but it won't allow buffer pooling.

Upvotes: 1

Views: 1166

Answers (1)

Marc Gravell
Marc Gravell

Reputation: 1062830

The question here is about the payload field. No, there is not current a mechanism to handle that, but it is certainly something that could be investigated for options. It could be that we can do something like an ArraySegment<byte> AllocateBuffer(int size) callback on the serialization-context that the caller could use to take control of the allocations (the nice thing about this is that protobuf-net doesn't actually work with ArraySegment<byte>, so it would be a purely incremental change that wouldn't impact any existing working code; if no callback is provided, we would presumably just allocate a flat buffer like it does currently). I'm open to other suggestions, but right now: no - it will allocate a byte[] internally.

Upvotes: 2

Related Questions