CODEWITHSUNDEEP
c#datatableparallel-processingparallel.foreachparallel.for
Sarath Subramanian
Sarath Subramanian

Reputation: 21361

Update outside loop

I have location of various csv files in strFilePath array (almost 1000 files which have 1 million records in each file). It takes a lot of time to read from files and merge all the data to single datatable. So I decided to go forward with Parallel processing.

CURRENT CODE 

DataTable dtMerge=new DataTable();
for(int i=0;i<strFilePath.Count;i++)
{
     Parallel.For(0, 3,m =>
     {
         clsNewClass objCls=new clsNewClass();
         DataTable dt=objCls.ReadCSV(strFilePath[m+i]);
     });
     m+=3;
}

The question is, how can I merge all the data from datatable dt to a global datatable dtMerge or can include all the results to a global variable dtMerge?

EXPECTED CODE TO WORK

DataTable dtMerge=new DataTable();
for(int i=0;i<strFilePath.Count;i++)
{
     Parallel.For(0, 3,m =>
     {
         clsNewClass objCls=new clsNewClass();
         // Is it possible like the below?
         dtMerge = objCls.ReadCSV(strFilePath[m+i]);
     });
     m+=3;
}

Upvotes: 1

Views: 160

Answers (1)

Scott Chamberlain
Scott Chamberlain

Reputation: 127603

Use the overload of Parallel.For (or ForEach) that gives you a local initializer and finally for the threads so you can merge your progress intra-thread without using locks. You can then merge your intra thread tables in the finally block in to your outside table using a lock for thread safety.

DataTable dtMerge = new DataTable();

Parallel.ForEach(strFilePath,
     () => new DataTable(),
     (filePath, loopState, local) =>
     {
         clsNewClass objCls=new clsNewClass();
         // Is it possible like the below?
         var dt = objCls.ReadCSV(filePath);
         local.Merge(dt, true, MissingSchemaAction.Add);
         return local;
     },
     (local) =>
     {
         lock(dtMerge)
         {
             dtMerge.Merge(local, true, MissingSchemaAction.Add);
         }
     });

I also got rid of your outer for loop, and replaced your inner loop with a parallel foreach there was no reason to nest your loops like that, just use a ForEach

Upvotes: 1

Related Questions