CODEWITHSUNDEEP
c#.netasynchronousparallel.foreach
eugeneK
eugeneK

Reputation: 11116

How can I limit Parallel.ForEach?

I have a Parallel.ForEach() async loop with which I download some webpages. My bandwidth is limited so I can download only x pages per time but Parallel.ForEach executes whole list of desired webpages.

Is there a way to limit thread number or any other limiter while running Parallel.ForEach?

Demo code:

Parallel.ForEach(listOfWebpages, webpage => {
  Download(webpage);
});

The real task has nothing to do with webpages, so creative web crawling solutions won't help.

Upvotes: 383

Views: 250254

Answers (5)

AliSalehi
AliSalehi

Reputation: 355

I think the more dynamic and realistic approach would be to limit it by the processor count, so on each system it would function properly:

var options = new ParallelOptions { MaxDegreeOfParallelism = Environment.ProcessorCount };
Parallel.ForEach(myList, options, iter => { });

perhpas yu would multiply Environment.ProcessorCount or divide it to put or take more pressure from the CPU

Upvotes: 5

rikitikitik
rikitikitik

Reputation: 2450

You can use ParallelOptions and set MaxDegreeOfParallelism to limit the number of concurrent threads:

Parallel.ForEach(
    listOfwebpages, 
    new ParallelOptions{MaxDegreeOfParallelism=2}, 
    webpage => {Download(webpage);});

Upvotes: 63

user3496060
user3496060

Reputation: 846

And for the VB.net users (syntax is weird and difficult to find)...

Parallel.ForEach(listOfWebpages, New ParallelOptions() With {.MaxDegreeOfParallelism = 8}, Sub(webpage)
......end sub)

Upvotes: 18

Richard
Richard

Reputation: 109005

Use another overload of Parallel.Foreach that takes a ParallelOptions instance, and set MaxDegreeOfParallelism to limit how many instances execute in parallel.

Upvotes: 25

Nick Butler
Nick Butler

Reputation: 24383

You can specify a MaxDegreeOfParallelism in a ParallelOptions parameter:

Parallel.ForEach(
    listOfWebpages,
    new ParallelOptions { MaxDegreeOfParallelism = 4 },
    webpage => { Download(webpage); }
);

MSDN: Parallel.ForEach

MSDN: ParallelOptions.MaxDegreeOfParallelism

Upvotes: 707

Related Questions