Import Excel file into database using Epplus and Sqlbulkcopy in MVC 5

Question

I am importing a large data file from excel into my database table using Epplus library in MVC5 and Entity Framework 6. It is taking alot of time to copy that file to the database. I am looking to use sqlbulkcopy to insert the data quickly but I am not really sure how to do that inside my controller. The code is working now but it takes alot of time for all the data to be imported into the database.

  public ActionResult Structure(FormCollection formCollection)
    {
        var usersList = new List<bomStructuredImportTgt>();
        if (Request != null)
        {
            HttpPostedFileBase file = Request.Files["UploadedFile"];
            if ((file != null) && (file.ContentLength > 0) && !string.IsNullOrEmpty(file.FileName))
            {
                string fileName = file.FileName;
                string fileContentType = file.ContentType;
                byte[] fileBytes = new byte[file.ContentLength];
                var data = file.InputStream.Read(fileBytes, 0, Convert.ToInt32(file.ContentLength));
                using (var package = new ExcelPackage(file.InputStream))
                {
                    var currentSheet = package.Workbook.Worksheets;
                    var workSheet = currentSheet.First();
                    var noOfCol = workSheet.Dimension.End.Column;
                    var noOfRow = workSheet.Dimension.End.Row;
                    for (int rowIterator = 2; rowIterator <= noOfRow; rowIterator++)
                    {
                        var user = new bomStructuredImportTgt();
                        user.ActualDate = Convert.ToDateTime(workSheet.Cells[rowIterator, 1].Value);
                        user.Description = workSheet.Cells[rowIterator, 2].Value?.ToString();
                        user.Level = Convert.ToInt32(workSheet.Cells[rowIterator, 3].Value);
                        user.ParentPartNumber = workSheet.Cells[rowIterator, 4].Value?.ToString();
                        user.PartNumber = workSheet.Cells[rowIterator, 5].Value?.ToString();
                        user.PartName = workSheet.Cells[rowIterator, 6].Value?.ToString();
                        user.HNS = workSheet.Cells[rowIterator, 7].Value?.ToString();
                        user.DWGSZ = workSheet.Cells[rowIterator, 8].Value?.ToString();
                        user.Part = workSheet.Cells[rowIterator, 9].Value?.ToString();
                        user.L1Quantity = Convert.ToInt32(workSheet.Cells[rowIterator, 10].Value);
                        user.ColorM = workSheet.Cells[rowIterator, 11].Value?.ToString();
                        user.ATTCD = workSheet.Cells[rowIterator, 12].Value?.ToString();
                        user.KD = workSheet.Cells[rowIterator, 13].Value?.ToString();
                        user.Sell = workSheet.Cells[rowIterator, 14].Value?.ToString();
                        user.PlGroup = workSheet.Cells[rowIterator, 15].Value?.ToString();
                        user.PL1 = workSheet.Cells[rowIterator, 16].Value?.ToString();
                        user.AT1 = workSheet.Cells[rowIterator, 17].Value?.ToString();
                        user.PL2 = workSheet.Cells[rowIterator, 18].Value?.ToString();
                        user.AT2 = workSheet.Cells[rowIterator, 19].Value?.ToString();
                        user.PL3 = workSheet.Cells[rowIterator, 20].Value?.ToString();
                        user.Plant = workSheet.Cells[rowIterator, 21].Value?.ToString();
                        user.SHRPCMINMAX = workSheet.Cells[rowIterator, 22].Value?.ToString();
                        usersList.Add(user);
                    }
                }
            }
        }
        using (Dev_Purchasing_New_ModelEntities excelImportDBEntities = new Dev_Purchasing_New_ModelEntities())
        {
            foreach (var item in usersList)
            {
                excelImportDBEntities.bomStructuredImportTgts.Add(item);
            }
            excelImportDBEntities.SaveChanges();
        }
        return View("Structure");
    }

Answer 1

IMHO, it's not a good idea to load huge excels via web site directly because:

1. It is not so easy to use BULK insert or BCP from ASP.
2. You'll make a user waits until process is finished.

What about another approaches:

1. use ETL (SSIS) to perform all loads to database and just copy user's excel to some share folder where SSIS package can read this file. After data is loaded you can notify user with mail and link to his/her data or use a web-site notification.
2. The same as the first but without a SSIS. Just copy file to some share that can be read by sql service and run BULK Insert with SQL command. But keep in mind that BULK insert works with csv files.
3. The same as the first two but instead bulk insert or bulk copy you can try OPENROWSET command to read excel but you'll require some drivers to be installed on SQL server side.