CODEWITHSUNDEEP
c#parallel-processing
Mehdi LAMRANI
Mehdi LAMRANI

Reputation: 11597

Sequentially Splitting the load on a Parallel.Foreach Loop

I have a million elements in a List to process.
Dropping them crudely into a Parallel.ForEach would just saturate the CPU.
Instead I split the Elements Master Lists into pieces and drop the Sublists into a parallel loop.

List<Element> MasterList = new List<Element>();
Populate(MasterList); // puts a Million elements into list;

//Split Master List into 100 Lists of 10.0000 elements each
List<List<Element>> ListOfSubLists = Split(MasterList,100);

foreach (List<Element> EL in ListOfSubLists )
{
   Parallel.ForEach(EL, E =>
   {
     // Do Stuff
   }

  //wait for all parallel iterations to end before continuing   
}

What is the best way for waiting for all parallel iteration to end before continuing to next iteration of the upper loop ?

Edit :
as some answers stated, "saturate the CPU" is not an accurate expression.
Actually I just want to limit the CPU usage, and avoid excessive load on it coming from this processing.

Upvotes: 3

Views: 4163

Answers (2)

David Pfeffer
David Pfeffer

Reputation: 39833

Parallel.ForEach will not saturate the CPU; it uses some intelligence to decide how many parallel threads to run simultaneously, with a max of 63.

See: Does Parallel.ForEach limits the number of active threads?

You can also set the max degree of parallelism if you want, by supplying a ParallelOptions like new ParallelOptions { MaxDegreeOfParallelism = 5 } as the second argument to Parallel.ForEach.

As a last point, Parallel.ForEach blocks until all of the iterations have completed. So your code, as written, works. You do not need to wait for the iterations to complete.

Upvotes: 4

Eoin Campbell
Eoin Campbell

Reputation: 44268

What do you mean "Saturate the CPU"

You can still throttle the Parallel foreach loop by supplying it with ParallelOptions One property of which is a MaxDegreesOfParallelism

That will allow you to go back to your single collection e.g.

Parallel.ForEach(
     collection,
     new ParallelOptions { MaxDegreeOfParallelism = 5},
     E => { DoStuff(E) }
);

Upvotes: 2

Related Questions