ProtoBuf-Net err msg - "An exception of type 'System.OutOfMemoryException' occurred in mscorlib.dll but was not handled in user code"

Question

While Serializing a dynamic object with around 250 properties and approx 20,000 rows I am getting the below error. The same code works fine when the number of properties are around 20. The error occurred at point Serializer.Serialize(stream, lst);

An unhandled exception of type 'System.OutOfMemoryException' occurred in System.ServiceModel.Internals.dll

at System.IO.MemoryStream.set_Capacity(Int32 value)
at System.IO.MemoryStream.EnsureCapacity(Int32 value)
at System.IO.MemoryStream.Write(Byte[] buffer, Int32 offset, Int32 count)
at ProtoBuf.ProtoWriter.Flush(ProtoWriter writer) in c:\Dev\protobuf-net\protobuf-net\ProtoWriter.cs:line 534
at ProtoBuf.ProtoWriter.Dispose() in c:\Dev\protobuf-net\protobuf-net\ProtoWriter.cs:line 478
at ProtoBuf.ProtoWriter.System.IDisposable.Dispose() in c:\Dev\protobuf-net\protobuf-net\ProtoWriter.cs:line 472
at ProtoBuf.Meta.TypeModel.Serialize(Stream dest, Object value, SerializationContext context) in c:\Dev\protobuf-net\protobuf-net\Meta\TypeModel.cs:line 218
at ProtoBuf.Meta.TypeModel.Serialize(Stream dest, Object value) in c:\Dev\protobuf-net\protobuf-net\Meta\TypeModel.cs:line 201
at ProtoBuf.Serializer.Serialize[T](Stream destination, T instance) in c:\Dev\protobuf-net\protobuf-net\Serializer.cs:line 87
at WcfService1.DynamicWrapper.Serialize(DynamicWrapper lst) in c:\Users\rkohli\Documents\Visual Studio 2012\Projects\WindowsFormsApplication3\WcfService1\SerializeObject.cs:line 136
at WcfService1.Service1.GetData(String sVisibleColumnList) in c:\Users\rkohli\Documents\Visual Studio 2012\Projects\WindowsFormsApplication3\WcfService1\Service1.svc.cs:line 22
at SyncInvokeGetData(Object , Object[] , Object[] )
at System.ServiceModel.Dispatcher.SyncMethodInvoker.Invoke(Object instance, Object[] inputs, Object[]& outputs)
at System.ServiceModel.Dispatcher.DispatchOperationRuntime.InvokeBegin(MessageRpc& rpc)
at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage5(MessageRpc& rpc)
at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage41(MessageRpc& rpc)
at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage4(MessageRpc& rpc)
at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage31(MessageRpc& rpc)
at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage3(MessageRpc& rpc)
at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage2(MessageRpc& rpc)
at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage11(MessageRpc& rpc)
at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage1(MessageRpc& rpc)
at System.ServiceModel.Dispatcher.MessageRpc.Process(Boolean isOperationContextSet)
at System.ServiceModel.Dispatcher.ChannelHandler.DispatchAndReleasePump(RequestContext request, Boolean cleanThread, OperationContext currentOperationContext)
at System.ServiceModel.Dispatcher.ChannelHandler.HandleRequest(RequestContext request, OperationContext currentOperationContext)
at System.ServiceModel.Dispatcher.ChannelHandler.AsyncMessagePump(IAsyncResult result)
at System.ServiceModel.Dispatcher.ChannelHandler.OnContinueAsyncReceive(Object state)
at System.Runtime.ActionItem.DefaultActionItem.TraceAndInvoke()
at System.Runtime.ActionItem.DefaultActionItem.Invoke()
at System.Runtime.ActionItem.CallbackHelper.InvokeWithoutContext(Object state)
at System.Runtime.IOThreadScheduler.ScheduledOverlapped.IOCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* nativeOverlapped)
at System.Runtime.Fx.IOCompletionThunk.UnhandledExceptionFrame(UInt32 error, UInt32 bytesRead, NativeOverlapped* nativeOverlapped)
at System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* pOVERLAP)

The below is the code sample.

[Serializable]
[ProtoContract]
public class DynamicWrapper
{
    [ProtoMember(1, DataFormat = DataFormat.Group)]
    public List<DictWrapper> Items { get; set; }

    public DynamicWrapper()
    {
        Items = new List<DictWrapper>();
    }

    public static byte[] Serialize(DynamicWrapper lst)
    {
        byte[] msgOut;

        using (var stream = new MemoryStream())
        {
            Serializer.Serialize(stream, lst);
            msgOut = stream.ToArray();
        }

        return msgOut;
    }

    public static DynamicWrapper Deserialize(byte[] message)
    {
        DynamicWrapper msgOut;

        using (var stream = new MemoryStream(message))
        {
            msgOut = Serializer.Deserialize<DynamicWrapper>(stream);
        }

        return msgOut;
    }
}

[Serializable]
[ProtoContract]
public class DictWrapper
{
    [ProtoMember(1, DataFormat = DataFormat.Group)]
    public Dictionary<string, string > Dictionary { get; set; }

    public DictWrapper()
    {
        Dictionary = new Dictionary<string, string>();
    }
}

Answer 1

There's nothing magic going on here. Based on the code, and a working project sent separately - the data for that is simply: big. 120446305 bytes, to be precise (based on the sample data). The main problem here is that you are using string property names, and duplicating them over and over and over and over and over. Now, protobuf-net can support string caching and re-use, but it doesn't do it by default - and there's no easy way to apply it to Dictionary<string,string>. But frankly, before I go figuring out crazy ways of making it work in this case (which would, by necessity, be a breaking change) I must first point out that this simply isn't a good fit for protobuf. Protobuf doesn't offer a "always smaller" guarantee: it offers to do a good job for typical scenarios, i.e. where your schema is known up-front and is predictable. Everything this one particular scenario is not.

Indeed, in the example given, it is loading data from a DataSet - which notably is only 284MB in the original data. Your use of protobuf-net here, for a scenario which is isn't aimed at, has resulted in a 4× growth in size.

Frankly, you'd have done better to send the original DataSet payload. Or even better: switch the data-set to binary mode and send that, (162 MB).

using (var file = File.Create("binary-ds"))
{
    dataSet.WriteXml(file, XmlWriteMode.WriteSchema);
}

Or better yet - switch it to binary mode and run it through gzip, for a sum total of 15MB:

using (var file = File.Create("binary-ds"))
using (var gzip = new GZipStream(file, CompressionLevel.Optimal))
{
    dataSet.WriteXml(gzip, XmlWriteMode.WriteSchema);
}

If I re-wrote your example using regular POCO/DTO classes and ran that through protobuf-net, I suspect the result would be similar (but without all the overhead of DataTable here), but there is not going to be a simple way of changing your scenario to play nicely with protobuf-net without having to change the data layout. And if you need to change the data layout - the above is a lot easier.