First called method faster than second one

Question

I had a look at the .NET Framework Sourcecode and stumbled on the implementation of LINQ-Sum

int Sum(this IEnumerable source)

I saw that it was implemented with a foreach-loop and wondered why the guys at MS don't use a normal for-loop due to performance reasons (later I learnt that there is no longer a performance difference between a for-loop and a foreach-loop - but I didn't know that just until now).

So I copied the MS-implementation in my own project and wrote a little benchmark:

var range = Enumerable.Range(1, 1000);
Stopwatch sw = new Stopwatch();

//Do sth unimportant for warming up

sw.Start();
for(int i = 0; i <= 10000; i++)
{
    long z = i + 3;
}
sw.Stop();

//Implementation 1

sw.Reset();
sw.Start();
for (int i = 0; i <= 1000000; i++)
{
    long i1 = range.Sum1();
}
sw.Stop();
Console.WriteLine("Sum1: " + sw.ElapsedTicks.ToString());

//Implementation 2

sw.Reset();
sw.Start();
for (int i = 0; i <= 1000000; i++)
{
    long i2 = range.Sum2();
}
sw.Stop();
Console.WriteLine("Sum2: " + sw.ElapsedTicks.ToString());

And here are the two implementations of Sum (Note: both are identical, I first wanted to check if the measuring is working correctly):

public static class LinqExtension
{
    public static int Sum1(this IEnumerable source)
    {
        int sum = 0;
        checked
        {
            foreach (int v in source) sum += v;
        }
        return sum;
    }

    public static int Sum2(this IEnumerable source)
    {
        int sum = 0;
        checked
        {
            foreach (int v in source) sum += v;
        }
        return sum;
    }
}

Surprisingly I got two different results : Sum1 = 16043441 vs. Sum2 = 17480907

So I extended the benchmark a little bit and called Sum1 and Sum2 not just once, but multiple times in the following order:

Sum1: 16035534
Sum2: 17381296
Sum2: 17441259
Sum1: 16021378
Sum1: 16000879
Sum1: 15989672
Sum2: 17342804
Sum2: 17347417 ...

Hence Sum1 is always nearly 10% faster than Sum2. When I call Sum2 first, the result are contrary.

What causes these performance differences? Why is the first called method faster than the second one? Is my benchmark invalid?

I'm using Visual Studio 2015 CTP4 and .NET Framework 4.5.3

EDIT:

Results in milliseconds instead of ticks

Sum1: 7714 ms
Sum2: 8336 ms
Sum2: 8321 ms
Sum1: 7686 ms
Sum1: 7693 ms
Sum1: 7686 ms
Sum2: 8372 ms
Sum2: 8302 ms ...

Thanks to the comments, I fixed some mistakes and now the code looks like that:

sw.Start();
for (int i = 0; i <= 1000000; i++)
{
   i1 = range.Sum1();
}
sw.Stop();
Console.WriteLine("Sum1: " + sw.ElapsedMilliseconds.ToString() + "
" + i1.ToString());

Now the results are totally different:

Sum1: 8021 ms
Sum2: 7587 ms
Sum2: 7660 ms
Sum1: 7989 ms
Sum1: 8041 ms
Sum1: 8038 ms
Sum2: 7609 ms
Sum2: 7613 ms

But there is still a difference, yet now the other way around.

Another update:

When I use

int[] range = new int[1000];
for (int m = 0; m < range.Length; m++)
            range[m] = m+1;

instead of

var range = Enumerable.Range(1, 1000);

both methods are equally fast.

Sum1: 6966 ms
Sum2: 6986 ms
Sum2: 7045 ms
Sum1: 7039 ms
Sum1: 6932 ms
Sum1: 7064 ms
Sum2: 7023 ms
Sum2: 7026 ms

Update: Tested it with Mono(SharpDevelop) and VS2013 and I got perfectly consistent results. So I think using VS2015 wasn't a great idea, since it's still a beta. Therefore the significance of the results is pretty low.

Another Update:

stakx commented:

Try calling each of your Sum1 and Sum2 methods at least once before you start measuring time, in order to make sure that the methods' code has been generated by the JIT. Otherwise you might be including the time required for JIT code generation in your benchmarking

So I called Sum1 and Sum2 one time before the measurings and suprisingly this solves the problem. But I don't understand why. I understand that generating the code by the JIT costs some time, but only the first time. In my test I have 20 for-loops, each of them calling Sum1 respectively Sum2 1.000.000 times. I do a measuring for every loop, and get constantly different values for Sum1 and Sum2. It would make sense, if the very first loop is slower, but that's not the case.

I've used ngen.exe to generate a native image and got the following results:

Sum1: 6517 ms
Sum2: 6837 ms
Sum2: 6817 ms
Sum1: 6511 ms
Sum1: 6513 ms
Sum1: 6513 ms
Sum2: 6822 ms
Sum2: 6942 ms ...

So there is still this difference.

Very important: It is NOT always the first method which is faster! Sometimes it's the first called method, sometimes the second one. But once the assembly was built the results are reproducible. It's pretty confusing for me and I can't see any pattern, when this happens.

Enigmativity:

Did you ever try swapping the order in which you called the methods? Calling Sum2 first?

Yeah, but then the result are just inverse. If Sum1 was the "fast method", after swapping, Sum2 is the fast one and Sum1 is the slow one.

First called method faster than second one

Answers (1)

Related Questions