CODEWITHSUNDEEP
c#.netlistserializationprotobuf-net
molnarriso
molnarriso

Reputation: 123

Weird Protobuf speedup by serializing list of objects

I have found a very weird issue with ProtoBuf performance regarding serialization of large number of complex objects. Lets have these two scenarios :

A) Serialize a list of objects one by one

B) Serialize the list as a whole

By intuition, this should have similar performance. However, in my application, there 10x difference in deserialization just by putting the objects in the list and serializing the list. Bellow you can find code to test this. The results vary in this example between 2x and 5x speedup, however in my code its prretty consistent 10x speedup.

What is causing this ? I have an app where I need to serialize objects one by one and its really downgrading performance, is there any way to increase performance of one by one serialization ?

Thanks

Output of code bellow

One by one serialization = 329204 ; deserialization = 41342
List serialization       = 19531 ; deserialization = 27716

Code

[ProtoContract]
    class TestObject
    {
        [ProtoMember(1)]public string str1;
        [ProtoMember(2)]public string str2;
        [ProtoMember(3)]public int i1;
        [ProtoMember(4)]public int i2;
        [ProtoMember(5)]public double d1;
        [ProtoMember(6)]public double d2;
        public TestObject(int cnt)
        {
            str1 = $"Hello World {cnt}";
            str2 = $"Lorem ipsum {cnt}";
            for (int i = 0; i < 2 ; i++) str1 = str1 + str1;
            d1 = i1 = cnt;
            d2 = i2 = cnt * 2;
        }
        public TestObject() { }
    }
    private void ProtoBufTest()
    {
        //init test data
        List<TestObject> objects = new List<TestObject>();
        int numObjects = 1000;
        for(int i = 0; i < numObjects;i++)
        {
            objects.Add(new TestObject(i));
        }
        Stopwatch sw = new Stopwatch();
        MemoryStream memStream = new MemoryStream();

        //test 1 
        sw.Restart();
        for (int i = 0; i < numObjects; i++)
        {
            ProtoBuf.Serializer.SerializeWithLengthPrefix<TestObject>(memStream, objects[i], ProtoBuf.PrefixStyle.Base128);
        }
        long timeToSerializeSeparately = sw.ElapsedTicks;
        memStream.Position = 0;

        sw.Restart();
        for (int i = 0; i < numObjects; i++)
        {
            ProtoBuf.Serializer.DeserializeWithLengthPrefix<TestObject>(memStream, ProtoBuf.PrefixStyle.Base128);
        }
        long timeToDeserializeSeparately = sw.ElapsedTicks;

        //test 2
        memStream.Position = 0;
        sw.Restart();
        ProtoBuf.Serializer.SerializeWithLengthPrefix<List<TestObject>>(memStream, objects, ProtoBuf.PrefixStyle.Base128);
        long timeToSerializeList = sw.ElapsedTicks;

        memStream.Position = 0;
        sw.Restart();
        ProtoBuf.Serializer.DeserializeWithLengthPrefix<List<TestObject>>(memStream,  ProtoBuf.PrefixStyle.Base128);
        long timeToDeserializeList = sw.ElapsedTicks;

        Console.WriteLine($"One by one serialization = {timeToSerializeSeparately} ; deserialization = {timeToDeserializeSeparately}");
        Console.WriteLine($"List serialization       = {timeToSerializeList} ; deserialization = {timeToDeserializeList}");
    }

Upvotes: 2

Views: 320

Answers (1)

Marc Gravell
Marc Gravell

Reputation: 1062865

I think you are misrepresenting the initial reflection pre-processing and the JIT costs; if we change it so it runs the test multiple times:

static void Main()
{
    ProtoBufTest(1);
    for (int i = 0; i < 10; i++)
    {
        ProtoBufTest(1000);
    }
}

private static void ProtoBufTest(int numObjects)
{
    ...

then I the results I would expect, where the single object code is faster.

Basically, it does a lot of work the first time it is needed, essentially exactly what you ask here:

is there some way to force ProtoBuf to cache reflection data between calls ? That would help a lot probably

already happens. As a side note, you can also do:

    Serializer.PrepareSerializer<TestObject>();

once right at the start of your app, and it will do as much as possible then. I can't force JIT to happen, though - to do that, you need to invoke the code once.

Upvotes: 2

Related Questions