Parallel.ForEach poor performance

Question

I wrote a little program that converts a bunch of files to pdf.

The program does the following:

• Get an Array of FileInfo objects from a Folder (10'000 docs)
• For each FileInfo
  • Create a backup copy with FileInfo.CopyTo(),
  • Convert the Document to PDF by using some Aspose Libraries
  • After conversion, copy the PDF to a new destination
  • Inside the foreach an Event is raised and handled by a WinForm UI to show some progress

Depending on the size of the Document the conversion of a Document can take 0-3 seconds. I thought that would be a perfect candidate for Parallel.ForEach, so I modified the program.

However the conversion took instead of 1 hour with conventional foreach 1.5 hours with Parallel.Foreach (The Server I've tried it has 2 x Intel Xeon Procs).

What did I do wrong or what do I need to consider to get better performance?

Answer 1

I can think on several problems that can cause the Parallel.Foreach to be slower:

1. Running more threads than processors.
2. Aspose Libraries isn't support multithread.
3. Multiple approaches to GUI thread that is threadsafe and cannot be reached from different threads at the same times.

also I recommend you to read my previous answer about Task parallel library - Parallelism on single core

It talks about single core but it can reflect on your problem.

Answer 2

It would depend on quite a few things. I would certainly try setting MaxDegreeOfParallelism to 2, in the hope that if the conversion is is CPU-bound and single-threaded, then having one per core should be close to ideal, though certainly experiment further.

But your very approach assumes that the conversion doesn't itself make good use of multiple cores. If it does, and it's CPU-bound, then it's already doing to sort of parallel use of cores that you are trying to introduce, and you're likely just going to make the whole thing less efficient for that reason.

Edit: Thought made clearer in light of svick's comment. If the library doesn't support multi-threaded use then it's unlikely to have gotten this far without erroring, but its support for multi-threading may involve a lot of internal locking that could be fine when there are occasional concurrent calls, but very expensive if you've got long-term heavy concurrency.

Answer 3

I recommend checking if your operation is CPU bound or I/O bound by looking at the CPU in taskmanager and Disk I/O response time/queue length in Resource Monitor and/or looking at the various available performance counters.

I suspect your problem is most likely that you are now doing multiple file copies (both for creating the backup and writing the converted file) at the same time. Hard disks are much faster for sequential access (if you only write/read one file at a time) compared to random access.