Insert huge number of rows into database using Entity Framework

Question

@EDIT I followed steps from Fastest Way of Inserting in Entity Framework and got even worse results, so it's not a duplicate.

My goal is to create a seed method to fill one of the tables of LocalDb. The method will add 182500 rows (to mimic yearly activity data for 500 devices) for further test. I might want to run it few more times to change the number of devices, so even more rows will be generated. That is why I need to insert rows as efficient as possible.

    protected void SeedReportDataTable(int numberOfTerminals)
    {
        var rand = new Random();
        var tidsList = new List<string>();

        // generuj liste losowych numerow tid
        for (int i = 0; i < numberOfTerminals; i++)
        {
            var randomTid = rand.Next(100000, 1000000).ToString(); // generuj 6-cyfrowy numer tid
            while (tidsList.Contains(randomTid)) { randomTid = rand.Next(100000, 1000000).ToString(); } // elminuj powtorzenia
            tidsList.Add(randomTid);
        }

        // dla kazdego z numerow tid generuj roczna historie aktywnosci
        var recordsList = new BlockingCollection<ReportData>();
        int year = Convert.ToInt32(DateTime.Now.Year);

        Parallel.ForEach(tidsList, tid =>
        {
            // dla kazdego miesiaca
            for (int month = 1; month <= 12; month++)
            {
                // dla kazdego dnia
                for (int day = 1; day <= DateTime.DaysInMonth(year, month); day++)
                {
                    var record = new ReportData
                    {
                        Tid = tid,
                        Active = Convert.ToBoolean(
                            rand.Next(0, 11)), // generuj losowy stan aktywnosci z prawdopodbienstwem 1/10 na bycie nieaktywnym
                        Date = new DateTime(year, month, day)
                    };
                    recordsList.Add(record);
                }
            }
        });
        // dodaj unikalne klucze glowne rekordom przed dodaniem do kontekstu bazy
        var keyValue = 1;

        foreach (var record in recordsList)
        {
            record.Id = keyValue++;
        }

        // podziel liste na czesci
        int chunkSize = 1000;

        for (int recordsSkipped = 0; recordsSkipped < recordsList.Count; recordsSkipped += chunkSize)
        {
            // wymieniaj kontekst
            using (var db = new dbEntities())
            {
                db.Configuration.AutoDetectChangesEnabled = false;
                db.Configuration.ValidateOnSaveEnabled = false;
                // dodawaj do bazy po kawalku
                db.ReportData.AddRange(recordsList.Skip(recordsSkipped).Take(chunkSize));
                db.SaveChanges();
            }
        }
    }

Running this code took 30 minutes to complete. Before that I ran a version ending with:

using (var db = new dbEntities())
{
    db.ReportData.AddRange(recordsList);
    db.SaveChanges();
}

and it took 15 minutes, which is still slower than I expected.

Why did my "improvements" fail?

What can I do to make it insert rows faster?

Answer 1

When I add my seeding method to Configuration.cs and run update-database command it takes less than 5 minutes to insert all rows.

It works best when calling Context.AddRange() only once.

        dbContext.Configuration.AutoDetectChangesEnabled = false;
        dbContext.Configuration.ValidateOnSaveEnabled = false;
        dbContext.ReportData.AddRange(recordsList);
        dbContext.SaveChanges();