CODEWITHSUNDEEP
.netmultithreading.net-4.5
Basic
Basic

Reputation: 26766

Correct way to manage threads when processing a queue

I've got a queue (actually 3 different queues but I don't believe that will make a difference). The queue contains discrete units of work.

I have a QueueManager which spawns a new thread and returns. The new thread looks at the queue, gets a piece of work and then spawns a new thread to process it.

I obviously want a limit on the number of concurrent threads and don't see any reason not to reuse a thread once it's finished the job.

Some of the jobs will take a while to run (minutes/hours) and I've read that the ThreadPool is not a good option for anything that takes more than a few seconds.

Effectively, I want to implement the ThreadPool but with a little more control.

So... What's the standard, best-practice way to achieve this?

It seems I need to (assuming the queue length > MaxThreadCount)

(The queue in this case is in a database so no event when items are added - I assume I'll just have to poll)

A lot of the jobs involve retrieving pages from remote sites/APIs, often repeatedly. As such, I believe I'd benefit from having considerably more threads than cores (as most of them will be waiting for the network). As such, I don't believe WaitHandle is appropriate as it has a 64-Thread limit.

This is obviously a pretty common pattern so I assume there must be a well-established way of implementing it?

If someone could point me at a good example, the would be excellent.

Upvotes: 0

Views: 160

Answers (1)

Chris Shain
Chris Shain

Reputation: 51369

Use the TPL (Task Parallel Library). It's designed for exactly this purpose.

http://msdn.microsoft.com/en-us/library/dd460717.aspx

Specifically, it sounds like you should create a TaskFactory with a TaskScheduler instance whose MaximumConcurrencyLevel has been set to the number of threads that you want. Then instead of putting your work items in a queue, call TaskFactory.StartNew. It will handle the queueing of work for you.

Upvotes: 2

Related Questions