CODEWITHSUNDEEP
c#performanceparallel-processingparallel.foreach
nobody
nobody

Reputation: 11080

Parallel.ForEach slower after upgrading server

I have daily process that runs on 20 folders with 60k+ files in it.There are no sub folders and the input files are few MB in size.For each folder, I read the files, parse it and write some data to a separate output file (i.e. 20 output files).I recently upgraded our server(higher cores & higher memory) and noticed a sharp decline in performance.I hope someone can point me the issue.

Below is my code

int iFolderCount = 0;
DirectoryInfo oSourceFolder = new DirectoryInfo(sInputFolder);
DirectoryInfo[] oIdDirectoryList = oSourceFolder.GetDirectories().Where(Id => sFolderList.Contains(Id.Name.ToUpper())).ToArray<DirectoryInfo>();
Parallel.ForEach(oIdDirectoryList, new ParallelOptions { MaxDegreeOfParallelism =  Environment.ProcessorCount }, (oId, state) =>
{
     FileInfo[] sFileList = oId.GetFiles();
     RawCounter.GetOrAdd(oId.Name.ToUpper(), sFileList.Length);

     using (StreamWriter oHandoffWriter = new StreamWriter(new FileStream(string.Format("{0}{1}_{2}_{3}{4}", sOutputFolder, Day, sOutputFileName, Interlocked.Increment(ref iFolderCount), HANDOFF_FILE_EXTENSION),FileMode.Append,FileAccess.Write,FileShare.Write)))
     {
         int iFileCounter = 0;
         foreach (FileInfo oFileInfo in sFileList)
         {
             try
             {
                 ProcessFile(oFileInfo, oHandoffWriter);
                 iFileCounter++;
             }
             catch (Exception ex)
             {
                 oLog.Info("Failed to process file " + oFileInfo.Name);
                 oLog.Info(ex.Message);
                 oLog.Info(ex.StackTrace);
                 oLog.Info(ex.InnerException);
                 File.Copy(oFileInfo.FullName, sErrorFileFolderPath + oFileInfo.Name, true);
             }
         }
         ProcessedCounter.GetOrAdd(oId.Name.ToUpper(), iFileCounter);
    }
});

After moving the process to the new server I noticed a sharp decline in performance.We went from 8 cores to 36 cores and 8GB RAM to 128 GB RAM.

Server Configuration

I started reducing the degree of parallelism and noticed that the process performed better everytime I reduced the degree parallelism.

MaxDegreeOfParallelism =  2

By setting it to 2 I am seeing higher performance.What am I missing here? MaxDegreeOfParallelism = Environment.ProcessorCount is faster on a older server running windows server 2008, 8 Core,8GB RAM while MaxDegreeOfParallelism = 2 is faster on a new windows server 2012,32 Core,128 GB RAM.

EDIT: I agree that the process is IO intensive.The daily file count/size does not vary much. Below are complete times for the process on the old server.

On the new server when I used Environment.ProcessorCount
- Handoff completed for day 20151118.The process took 712.05125171 minutes.

On the new server when I used 2 as the degree of parallelism
- Handoff completed for day 20151118.The process took 89.61782427 minutes.

I'll do more runs and update this question.

Upvotes: 2

Views: 170

Answers (1)

usr
usr

Reputation: 171178

The processing is neither CPU nor memory bound so that the new server does not help at all. It seems IO bound. Did the IO hardware change? Parallelism plus IO can easily mean slower performance because sequential IO might be turned into random IO.

It is a mistake to choose the DOP for IO based on the number of CPU cores. The optimal IO DOP is unrelated to the core count. Determine the optimal DOP empirically.

Upvotes: 4

Related Questions