CODEWITHSUNDEEP
c#xml
petko_stankoski
petko_stankoski

Reputation: 10713

Reading large xml file makes the server stop working - out of memory

I have a piece of code which works well for normal files. But for really big files, it makes the server stop working.

Here it is:

XmlReader reader = null;
try
{
    reader = XmlReader.Create(file_name + ".xml");
    XDocument xml = XDocument.Load(reader);
    XmlNamespaceManager namespaceManager = GetNamespaceManager(reader);
    XElement root = xml.Root;

    //XAttribute supplier = root.XPathSelectElement("//sh:Receive/sh:Id", namespaceManager).Attribute("Authority");

    //string version = root.XPathSelectElement("//sh:DocumentId/sh:Version", namespaceManager).Value;

    var nodes = root.XPathSelectElements("//eanucc:msg/eanucc:transact", namespaceManager);

    return nodes;                
}
catch
{ }

I think this is the part which causes the memory problem which happens on the server. How can I fix this?

Upvotes: 1

Views: 302

Answers (2)

Jon Skeet
Jon Skeet

Reputation: 1500485

It sounds like there's simply too much data to read in one go. You'll have to iterate over the elements one at a time, using XmlReader as a cursor, and converting one element to XElement at a time.

public static IEnumerable<XElement> ReadTransactions()
{
    using (var reader = XmlReader.Create(file_name + ".xml"))
    {
        while (reader.ReadToFollowing("transact", eanuccNamespaceUri))
        {
            using (var subtree = reader.ReadSubtree())
            {
                yield return XElement.Load(subtree);
            }
        }
    }
}

Note: this assumes there are never "transact" elements at any other level. If there are, you'll need to be more careful with your XmlReader than just calling ReadToFollowing. Also note that you'll need to find the actual namespace URI of the eanucc alias.

Don't forget that if you try to read all of this information in one go (e.g. by calling ToList()) then you'll still run out of memory. You need to stream the information. (It's not clear what you're trying to do with the elements, but you need to think about it carefully.)

Upvotes: 3

Jras
Jras

Reputation: 518

Try putting the reader in a using(){} clause so it gets disposed of after use.

try
{
    using(var reader = XmlReader.Create(file_name + ".xml"))
    {
      XDocument xml = XDocument.Load(reader);
      XmlNamespaceManager namespaceManager = GetNamespaceManager(reader);
      XElement root = xml.Root;

      var nodes = root.XPathSelectElements("//eanucc:msg/eanucc:transact", namespaceManager);

      return nodes;
    }                
}
catch
{ }

Upvotes: 0

Related Questions