CODEWITHSUNDEEP
c#xmlserializationout-of-memory
Adrian Aldea
Adrian Aldea

Reputation: 57

XML Serialize - BIG Data ( 20GB ), OutOfMemoryException

I have a problem: I want to serialize an xml ( 20GB ) but I get an out of memory exception.

Do you have any suggestion for this ?

The code I had is below:

public static string Serialize(object obj)
{
    string retval = string.Empty;

    if (null!= obj)
    {
        StringBuilder sb = new StringBuilder();
        using (XmlWriter writer = XmlWriter.Create(sb, new XmlWriterSettings() { OmitXmlDeclaration = true }))
        {                    
            XmlSerializer serializer = new XmlSerializer(obj.GetType());

            // We are ommitting the namespace to simplifying passing as parameter
            XmlSerializerNamespaces ns = new XmlSerializerNamespaces();
            ns.Add("", "");

            serializer.Serialize(writer, obj);
        }

        retval = sb.ToString();
    }
    return retval;
}

Upvotes: 1

Views: 1406

Answers (2)

jdweng
jdweng

Reputation: 34421

You probably have a List to do it in pieces

        public static void Serialize(List<MyClass> myClasses)
        {
            string retval = string.Empty;

            if (myClasses != null)
            {

                using (StreamWriter sWriter = new StreamWriter("filename", false))
                {
                    foreach (MyClass myClass in myClasses)
                    {

                        StringBuilder sb = new StringBuilder();
                        using (XmlWriter writer1 = XmlWriter.Create(sb, new XmlWriterSettings() { OmitXmlDeclaration = true }))
                        {
                            XmlSerializer serializer = new XmlSerializer(myClass.GetType());

                            // We are ommitting the namespace to simplifying passing as parameter
                            XmlSerializerNamespaces ns = new XmlSerializerNamespaces();
                            ns.Add("", "");

                            serializer.Serialize(writer1, myClass);
                        }
                        sWriter.Write(sb.ToString());

                    }

                }
            }
        }

Upvotes: 0

Marc Gravell
Marc Gravell

Reputation: 1062620

20 GB is never going to work as a string (via StringBuilder); even with <gcAllowVeryLargeObjects> enabled, the maxiumum theoretical length of a string is a tiny fraction of that.

If you want huge data, you need to use something like a file (or basically: a Stream that isn't MemoryStream) as the backend.

I would also say that xml is a terrible choice for large data. If you aren't tightly bound to xml, I would strongly recommend looking at alternative tools (I'm happy to offer advice, if that is an option).

But for now:

string path = "my.xml";
XmlWriterSettings settings = ...
using (XmlWriter writer = XmlWriter.Create(path, settings))
{
    // ...
}

or if you're actually talking to a socket etc:

Stream stream = ...
XmlWriterSettings settings = ...
using (XmlWriter writer = XmlWriter.Create(stream, settings))
{
    // ...
}

Upvotes: 3

Related Questions