c# performance changes rapidly by looping just 0,001% more often

Question

I'm working on an UI project which has to work with huge datasets (every second 35 new values) which will then be displayed in a graph. The user shall be able to change the view from 10 Minutes up to Month view. To archive this I wrote myself a helper function which truncate a lot of data to a 600 byte array which then should be displayed on a LiveView Chart.

I found out that at the beginning the software works very well and fast, but as longer the software runs (e.g. for a month) and the memory usage raises (to ca. 600 mb) the function get's a lot of slower (up to 8x).

So I made some tests to to find the source of this. Quite surprised I found out that there is something like a magic number where the function get's 2x slower , just by changing 71494 loops to 71495 from 19ms to 39ms runtime

I'm really confused. Even when you comment out the second for loop (where the arrays are geting truncated) it is a lot of slower. Maybe this has something to do with the Garbage Collector? Or does C# compress memory automatically?

Using Visual Studio 2017 with newest updates.

The Code

using System;
using System.Collections.Generic;
using System.Diagnostics;

namespace TempoaryTest
{
    class ProductNameStream
    {
        public struct FileValue
        {

            public DateTime Time;
            public ushort[] Value;
            public ushort[] Avg1;
            public ushort[] Avg2;
            public ushort[] DAvg;
            public ushort AlarmDelta;
            public ushort AlarmAverage;
            public ushort AlarmSum;
        }
    }


    public static class Program

    {
        private const int MAX_MEASURE_MODEL = 600;

        private const int TEST = 71494;
        //private const int TEST = 71495;//this one doubles the consuming time!
        public static void Main(string[] bleg)
        {
            List<ProductNameStream.FileValue> fileValues = new List<ProductNameStream.FileValue>();
            ProductNameStream.FileValue fil = new ProductNameStream.FileValue();

            DateTime testTime = DateTime.Now;

            Console.WriteLine("TEST: {0} {1:X}", TEST, TEST);
            //Creating example List
            for (int n = 0; n < TEST; n++)
            {
                fil = new ProductNameStream.FileValue
                {
                    Time = testTime = testTime.AddSeconds(1),
                    Value = new ushort[8],
                    Avg1 = new ushort[8],
                    Avg2 = new ushort[8],
                    DAvg = new ushort[8]
                };
                for (int i = 0; i < 8; i++)
                {
                    fil.Value[i] = (ushort)(n + i);
                    fil.Avg1[i] = (ushort)(TEST - n - i);
                    fil.Avg2[i] = (ushort)(n / (i + 1));
                    fil.DAvg[i] = (ushort)(n * (i + 1));
                }
                fil.AlarmDelta = (ushort)DateTime.Now.Ticks;
                fil.AlarmAverage = (ushort)(fil.AlarmDelta / 2);
                fil.AlarmSum = (ushort)(n);
                fileValues.Add(fil);
            }


            var sw = Stopwatch.StartNew();

            /* May look like the same as MAX_MEASURE_MODEL but since we use int
             *  as counter we must be aware of the int round down.*/
            int cnt = (fileValues.Count / (fileValues.Count / MAX_MEASURE_MODEL)) + 1;

            ProductNameStream.FileValue[] newFileValues = new ProductNameStream.FileValue[cnt];
            ProductNameStream.FileValue[] fileValuesArray = fileValues.ToArray();


            //Truncate the big list to a 600 Array
            for (int n = 0; n < fileValues.Count; n++)
            {
                if ((n % (fileValues.Count / MAX_MEASURE_MODEL)) == 0)
                {
                    cnt = n / (fileValues.Count / MAX_MEASURE_MODEL);
                    newFileValues[cnt] = fileValuesArray[n];
                    newFileValues[cnt].Value = new ushort[8];
                    newFileValues[cnt].Avg1 = new ushort[8];
                    newFileValues[cnt].Avg2 = new ushort[8];
                    newFileValues[cnt].DAvg = new ushort[8];

                }

                else
                {
                    for (int i = 0; i < 8; i++)
                    {
                        if (newFileValues[cnt].Value[i] < fileValuesArray[n].Value[i])
                            newFileValues[cnt].Value[i] = fileValuesArray[n].Value[i];
                        if (newFileValues[cnt].Avg1[i] < fileValuesArray[n].Avg1[i])
                            newFileValues[cnt].Avg1[i] = fileValuesArray[n].Avg1[i];
                        if (newFileValues[cnt].Avg2[i] < fileValuesArray[n].Avg2[i])
                            newFileValues[cnt].Avg2[i] = fileValuesArray[n].Avg2[i];
                        if (newFileValues[cnt].DAvg[i] < fileValuesArray[n].DAvg[i])
                            newFileValues[cnt].DAvg[i] = fileValuesArray[n].DAvg[i];
                    }
                    if (newFileValues[cnt].AlarmSum < fileValuesArray[n].AlarmSum)
                        newFileValues[cnt].AlarmSum = fileValuesArray[n].AlarmSum;
                    if (newFileValues[cnt].AlarmDelta < fileValuesArray[n].AlarmDelta)
                        newFileValues[cnt].AlarmDelta = fileValuesArray[n].AlarmDelta;
                    if (newFileValues[cnt].AlarmAverage < fileValuesArray[n].AlarmAverage)
                        newFileValues[cnt].AlarmAverage = fileValuesArray[n].AlarmAverage;
                }
            }
            Console.WriteLine(sw.ElapsedMilliseconds);
        }
    }
}

Answer 1

Your data structure is inefficient and is forcing you to do a lot of allocations during computation. Have a look of thisfixed size array inside a struct . Also preallocate the list. Don't rely on the list to constantly adjust its size which also creates garbage.

Answer 2

This is most likely being caused by the garbage collector, as you suggested.

I can offer two pieces of evidence to indicate that this is so:

1. If you put GC.Collect() just before you start the stopwatch, the difference in times goes away.
2. If you instead change the initialisation of the list to new List<ProductNameStream.FileValue>(TEST);, the difference in time also goes away.

(Initialising the list's capacity to the final size in its constructor prevents multiple reallocations of its internal array while items are being added to it, which will reduce pressure on the garbage collector.)

Therefore, I assert based on this evidence that it is indeed the garbage collector that is impacting your timings.

Incidentally, the threshold value was slightly different for me, and for at least one other person too (which isn't surprising if the timing differences are being caused by the garbage collector).