CODEWITHSUNDEEP
multithreadingc#-4.0task-parallel-library
GrandMasterFlush
GrandMasterFlush

Reputation: 6409

C# TPL Tasks - How many at one time

I'm learning how to use the TPL for parellizing an application I have. The application processes ZIP files, exctracting all of the files held within them and importing the contents into a database. There may be several thousand zip files waiting to be processed at a given time.

Am I right in kicking off a separate task for each of these ZIP files or is this an inefficient way to use the TPL?

Thanks.

Upvotes: 7

Views: 1814

Answers (4)

ipavlu
ipavlu

Reputation: 1649

I have to disagree with certain statements here guys.

First of all, I do not see any difference between ThreadPool and Tasks in coordination or control. Especially when tasks runs on ThreadPool and you have easy control over tasks, exceptions are nicely propagated to the caller during await or awaiting on Tasks.WhenAll(tasks) etc.

Second, I/O wont have to be the only bottleneck here, depending on data and level of compression the ZIPping is going to take msot likely more time than reading the file from the disc.

It can be thought of in many ways, but I would best go for something like number of CPU cores or little less.

Loading file paths to ConcurrentQueue and then allowing running tasks to dequeue filepaths, load files, zip them, save them.

From there you can tweak the number of cores and play with load balancing.

I do not know if ZIP supports file partitioning during compression, but in some advanced/complex cases it could be good idea especially on large files...

WOW, it is 6 years old question, bummer! I have not noticed...:)

Upvotes: 0

Paul Sasik
Paul Sasik

Reputation: 81449

This seems like a problem better suited for worker threads (separate thread for each file) managed with the ThreadPool rather than the TPL. TPL is great when you can divide and conquer on a single item of data but your zip files are treated individually.

Disc I/O is going to be your bottle neck so I think that you will need to throttle the number of jobs running simultaneously. It's simple to manage this with worker threads but I'm not sure how much control you have (if nay) over the parallel for, foreach as far as how parallelism goes on at once, which could choke your process and actually slow it down.

Upvotes: 4

Hans Olsson
Hans Olsson

Reputation: 55009

I would have thought that this would depend on if the process is limited by CPU or disk. If the process is limited by disk I'd thought that it might be a bad idea to kick off too many threads since the various extractions might just compete with each other.

This feels like something you might need to measure to get the correct answer for what's best.

Upvotes: 1

Mitchel Sellers
Mitchel Sellers

Reputation: 63126

Anytime that you have a long running process, you can typically gain additional performance on multi-processor systems by making different threads for each input task. So I would say that you are most likely going down the right path.

Upvotes: 1

Related Questions