OutOfMemoryException receiving data from WebServices

Question

I have an application that communicates with Java webservices using WCF clients. One of the services returns a moderately big result (about 100 Mb) and sometimes we get an OutOfMemoryException:

System.IO.MemoryStream.set_Capacity(Int32)
System.IO.MemoryStream.EnsureCapacity(Int32)
System.IO.MemoryStream.Write(Byte[], Int32, Int32)
System.Xml.XmlMtomReader+MimePart.GetBuffer(Int32, Int32 ByRef)
System.Xml.XmlMtomReader.Initialize(System.IO.Stream, System.String, System.Xml.XmlDictionaryReaderQuotas, Int32)
System.Xml.XmlMtomReader.SetInput(System.IO.Stream, System.Text.Encoding[], System.String, System.Xml.XmlDictionaryReaderQuotas, Int32, System.Xml.OnXmlDictionaryReaderClose)
System.ServiceModel.Channels.MtomMessageEncoder.TakeStreamedReader(System.IO.Stream, System.String)
System.ServiceModel.Channels.MtomMessageEncoder.ReadMessage(System.IO.Stream, Int32, System.String)
System.ServiceModel.Channels.HttpInput.ReadStreamedMessage(System.IO.Stream) System.ServiceModel.Channels.HttpInput.ParseIncomingMessage(System.Exception ByRef)
System.ServiceModel.Channels.HttpChannelFactory+HttpRequestChannel+HttpChannelRequest.WaitForReply(System.TimeSpan) System.ServiceModel.Channels.RequestChannel.Request(System.ServiceModel.Channels.Message, System.TimeSpan)
System.ServiceModel.Dispatcher.RequestChannelBinder.Request(System.ServiceModel.Channels.Message, System.TimeSpan)
System.ServiceModel.Channels.ServiceChannel.Call(System.String, Boolean, System.ServiceModel.Dispatcher.ProxyOperationRuntime, System.Object[], System.Object[], System.TimeSpan)
System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(System.Runtime.Remoting.Messaging.IMethodCallMessage, System.ServiceModel.Dispatcher.ProxyOperationRuntime)
System.ServiceModel.Channels.ServiceChannelProxy.Invoke(System.Runtime.Remoting.Messaging.IMessage) System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(System.Runtime.Remoting.Proxies.MessageData ByRef, Int32)

Our WCF Client ....

The amount of data is not big enough to create a real OutOfMemoryException as the application is a 32 bit app consuming about 400 - 600 MB and the response is aroung 100Mb so something else must be happening.

Any idea?

Answer 1

Try using a the WCF WebServiceHost and setting the configuration to not limit buffer or Message size:

    using System;
    using System.ServiceModel.Web;

    private WebServiceHost webHost;

    public void Start()
    {
        webHost.Opening += ConfigureEnpointBinding;

        webHost.Open();
    }

    private void ConfigureEnpointBinding(object sender, EventArgs e)
    {
        var endpointBinding = (System.ServiceModel.WebHttpBinding)
            ((WebServiceHost)sender)
            .Description
            .Endpoints
            .Single(endpoint => endpoint.Contract.ContractType == typeof(IYourInterface))
            .Binding;

        endpointBinding.MaxReceivedMessageSize = int.MaxValue;
        endpointBinding.MaxBufferSize = int.MaxValue;
    }

Answer 2

There are couple of things that lead to such behavior:

1. 32bit process can only address 2GB (sometimes 3GB) of it's virtual address space (1-2GB is reserved, and with 32-bit pointer you can only address 4GB).

2. .NET stores large objects (which are bigger than 85000 bytes) on a special Large Object Heap. This heap is by default not compacted (and in .NET versions before 4.5.1 - never compacted at all). Suppose you allocated 40MB, then 10MB then 70MB of memory. After a while, 40MB and 70MB were garbage collected. Now you want to allocate 100MB chunk. If Large Object Heap were compacted - you could have at least 110MB of continuous free address space. But it's not and so you have two gaps of 40MB and 70MB and you cannot allocate your 100MB chunk because of this.

So even if there is plenty free physical RAM on that machine (and even if not - there is always swap), you may not be able to get a pointer to continuous chunk of address space large enough. In this case OutOfMemoryException would be thrown.

Couple of ways to solve this:

1. Use 64-bit process.
2. Reuse (pool) memory chunks by reusing byte arrays. For example WCF provides (and uses) such BufferManager.

3. .NET 4.5.1 provides a way to compact Large Object Heap, but it does not really solves your problem I think, just kind of delays it. You can compact LOH once (you cannot force it to compact on every collection) by doing:

   GCSettings.LargeObjectHeapCompactionMode = GCLargeObjectHeapCompactionMode.CompactOnce;
   GC.Collect();