CODEWITHSUNDEEP
c#parallel-processingparallel.foreach
bilpor
bilpor

Reputation: 3889

How do I parallelize an IEnumerable of IEnumerable?

I have a variable:

var contexts where after setting is an IEnumerable. One of the properties on this object is also an IEnumerable I have:

Parallel.ForEach(contexts.AsParallel(), (context) =>
{
    Parallel.ForEach(context, (child) =>
    {

    });
});

but VS complains about the second ForEach.

Upvotes: 2

Views: 818

Answers (1)

StuartLC
StuartLC

Reputation: 107237

You're trying to apply way too much nested parallelism here.

Parallel.ForEach(   <-- #1
     contexts.AsParallel(), <-- #2
         (context) =>  {
        Parallel.ForEach( <- #3

Assuming this is CPU bound work, you really don't want to be doing this. This will potentially create too many Tasks which will need to be mapped onto threads, which will cause lots of counter-productive context switching.

Assuming a sufficient number of items at the outermost loop with equal volumes, then it is better to just keep the parallelism at the outer loop:

Parallel.ForEach(
  contexts, 
  context => { ...

(And if the number of items on the outer loop is insufficient to keep all cores busy, or if the item dispersion in the inner IEnumerable isn't even and will cause tasks to complete at unequal times, then consider using @TheGeneral's idea of using contexts.SelectMany(c => c), i.e. of the form outer.SelectMany(o => o.Inner), to flatten the nested collections into one larger one, before parallelising).

If you still find that concurrency is too high for CPU-bound work, you can match it to the number of actual cores on the target computer, e.g. if you have 8 cores available:

Parallel.ForEach(
  contexts, 
  new ParallelOptions { MaxDegreeOfParallelism = 8 },  
  context => { ...

However, if this is to be used for IO bound work, then Parallel.For* / .AsParallel are the wrong tools - use an asynchronous parallelism tool such as Task.WhenAll.

Upvotes: 4

Related Questions