CODEWITHSUNDEEP
c#asynchronousparallel-processing
wisbucky
wisbucky

Reputation: 37837

How to implement concurrent calls in C# console app, .NET 4.5?

I'm polling an external queue for job requests, then process the job (which takes ~1 minute). Currently, I'm doing it synchronously, which is inefficient because if there were 10 jobs, it would take 10 minutes. The machine can handle up to 5 jobs at the same time, so it should be able to process 10 jobs in ~2 minutes.

I've never done multi-threading, I've tried to read about async, await, Task.Run, but haven't had any success implementing it. Any suggestions on a simple way to implement this pseudocode?

while(concurrentJobs <= 5)
{
    job = pollForJob(); // synchronous

    processJob(job); // want to do this in background so that I can poll for next job
}

void processJob(job)
{
    concurrentJobs++;
    doStuff(); // takes 1 minute
    concurrentJobs--;
}

Upvotes: 2

Views: 1682

Answers (3)

R&#244;mulo Spier
R&#244;mulo Spier

Reputation: 231

Have you ever tried the parallel for each? It's simple.

Here a MSDN example

Parallel.ForEach(concurrentJobs , currentJob =>
{
    processJob(currentJob);
});

Upvotes: 0

oleksii
oleksii

Reputation: 35905

Looks like you have a producer-consumer model. In this case Parallel class won't do you any good, because a new job can come any time. You can only use Parallel class when you know beforehand what you need to do.

If, however, a new job can come any time I would use the following approach.

Producer

Write a separate long-running Task or use Thread. This thread receives jobs from network (such as from a queue or a database) or whatever and puts items into the shared memory.

Shared memory

Use BlockingCollection to add and take jobs. This acts as a mediator between producer and consumer.

Consumer

Exactly the same as producer - a separate Thread that checks the shared memory for any available item. Once an item is retrieved from a queue you can process it at will.

Now if you have such separation, you can easily spawn several producers and consumers, each running in its own thread.

Upvotes: 3

Scott Chamberlain
Scott Chamberlain

Reputation: 127563

Similar to Romulo's answer, but a little more fleshed out.

First you need to turn your pollForJob() function to an IEnumerable

private IEnumerable<Job> JobEnumerator()
{
   Job job;
   while((job = pollForJob()) != null)
   {
       yield return job;
   }
}

Then you can use it with the Parallel.ForEach class

Parallel.ForEach(JobEnumerator(), 
         new ParallelOptions() {MaxDegreeOfParallelism = 5}, 
         processJob);

This will block on the line Parallel.ForEach but it will run 5 instances of the processJob(Job job) function at the same time.

Upvotes: 0

Related Questions