Parallel.ForEach no thread at the end

Question

i'm testing an application which should compile much projects/files.

I've got a ConucrrentBag which should be worked through with Parallel.

private readonly ConcurrentBag<string> m_files;

My Call for Parallel is this:

Parallel.ForEach(m_files, new ParallelOptions
            {
                MaxDegreeOfParallelism = MaxProcesses,

            }, currFile => ProcessSingle(currFile.ToString()));

The Amount of MaxProcess is LogicalCpu*2.

When I'm compiling 140 Projects, to the end Parallel will start linear less threads. At least there's only one Thread running for the last 4 Projects. That's not nice, but okay.

Now my Problem:

When I'm compiling about 14000+ Projects (It's COBOL-SOURCE ;-) and a really big system) The last Modules won't be compiled, because Parallel.ForEach isn't starting new Threads for this. There is no Workingthread alive at this point. But there are still 140 items in the concurrentBag.

Anybody an idea how to solve this?

Edit: That Problem occurs only, when I run the compiler. Without running compiler (for faster testing) It work's fine...

Edit:

The ConcurrentBag is already filled completly when I start the Parallel.ForEach process.

For detailed information, the Code in SingleProcess:

private void ProcessSingle(string item)
        {
            Monitor.Enter(lockingObj);
            if (m_files.TryTake(out item))
            {
                if (CompilingModules <= 0)
                {
                    OnQueueStarted(new EventArgs());
                }
                CompilingModules++;
                Monitor.Exit(lockingObj);
                OnQueueItemStateChanged(new ItemQueueEventArgs(item, null, ItemQueueType.Done, ItemQueueObject.String));

                OnQueueItemStateChanged(new ItemQueueEventArgs(item, null, ItemQueueType.Dequeued, ItemQueueObject.String));
                using (CobolCompiler compiler = new CobolCompiler())
                {
                    compiler.OutputDataReceived += (sender, e) => OnOutputDataReceived(e);
                    compiler.Compile(item);
                    Thread.Sleep(2000);
                    if (compiler.LinkFailure)
                    {
                        if (ObjWithoutDll.ContainsKey(item))
                        {
                            if (ObjWithoutDll[item] <= 2)
                            {
                                m_files.Add(item);
                                OnQueueItemStateChanged(new ItemQueueEventArgs(item, null, ItemQueueType.Enqueued, ItemQueueObject.String));
                                ObjWithoutDll[item]++;
                            }
                            else
                            {
                                OnQueueItemStateChanged(new ItemQueueEventArgs(item, null, ItemQueueType.LinkError, ItemQueueObject.String));
                                ObjWithoutDll.Remove(item);
                            }
                        }
                        else
                        {
                            ObjWithoutDll.Add(item, 0);
                            m_files.Add(item);
                            OnQueueItemStateChanged(new ItemQueueEventArgs(item, null, ItemQueueType.Enqueued, ItemQueueObject.String));
                        }
                    }
                    else
                    {
                        if (compiler.DllExisting)
                        {
                            ObjWithoutDll.Remove(item);
                        }
                        OnQueueItemStateChanged(compiler.DllExisting ? new ItemQueueEventArgs(item, null, ItemQueueType.Done, ItemQueueObject.String) : new ItemQueueEventArgs(item, null, ItemQueueType.Failed, ItemQueueObject.String));
                    }

                }

                Monitor.Enter(lockingObj);
                CompiledModules++;
                if (CompiledModules % 300 == 0)
                {
                    Thread.Sleep(60000);
                }
                CompilingModules--;
                if (CompilingModules <= 0 && m_files.Count <= 0)
                {

                    try
                    {
                        Process prReschk = new Process();
                        FileInfo batch = new FileInfo(@"batches\reschkdlg.cmd");
                        if (!batch.Exists)
                        {
                            Assembly _assembly = Assembly.GetExecutingAssembly();
                            StreamReader _textStreamReader = new StreamReader(_assembly.GetManifestResourceStream(@"Batches\reschkdlg.cmd"));
                        }

                        if (!File.Exists(Config.Instance.WorkingDir + @"reschkdlg.exe"))
                        {
                            File.Copy(Config.Instance.VersionExeDirectory + @"reschkdlg.exe", Config.Instance.WorkingDir + @"reschkdlg.exe");
                        }

                        prReschk.StartInfo.FileName = @"cmd.exe";
                        prReschk.StartInfo.Arguments = @"/c " + batch.FullName + " " + Config.Instance.Version.Replace(".", "") + " " + @"*" + " " + Config.Instance.WorkingDir;
                        prReschk.StartInfo.CreateNoWindow = true;
                        prReschk.StartInfo.UseShellExecute = false;
                        prReschk.Start();
                        prReschk.Close();
                        prReschk.Dispose();
                    }
                    catch
                    {
                    }

                    OnQueueFinished(new EventArgs());
                }
            }
            Monitor.Exit(lockingObj);
        }

Here the Codesnippet of the CobolCompiler Class:

public void Compile(string file) {

        file = file.ToLower();

        Process prCompile = new Process();
        Dir = Directory.CreateDirectory(c.WorkingDir + random.Next() + "\\");

        try
        {
            // First clean up the folder
            CleanUpFolder(true, file);

            // First set lock and copy all sources
            Monitor.Enter(lockingObj);
            if (filesToCopy == null)
            {
                CopySource(Dir.FullName);
            }
            Monitor.Exit(lockingObj);

            FileInfo batch = new FileInfo(@"batches\compile.cmd");
            if (!batch.Exists)
            {
                Assembly _assembly = Assembly.GetExecutingAssembly();
                StreamReader _textStreamReader = new StreamReader(_assembly.GetManifestResourceStream(@"Batches\compile.cmd"));
                _textStreamReader.Dispose();
            }

            prCompile.StartInfo.FileName = @"cmd.exe";
            prCompile.StartInfo.Arguments = @"/c " + batch.FullName + " " + c.Version.Replace(".", "") + " " + file.Remove(file.LastIndexOf('.')) + " " + Dir.FullName + " " + Dir.FullName.Remove(Dir.FullName.IndexOf(@"\"));
            prCompile.StartInfo.CreateNoWindow = true;
            prCompile.StartInfo.UseShellExecute = false;
            prCompile.StartInfo.RedirectStandardOutput = true;
            prCompile.StartInfo.RedirectStandardError = true;
            prCompile.StartInfo.WorkingDirectory = Assembly.GetExecutingAssembly().Location.Remove(Assembly.GetExecutingAssembly().Location.LastIndexOf("\\") + 1);
            prCompile.EnableRaisingEvents = true;
            prCompile.OutputDataReceived += prCompile_OutputDataReceived;
            prCompile.ErrorDataReceived += prCompile_OutputDataReceived;
            prCompile.Start();
            prCompile.BeginErrorReadLine();
            prCompile.BeginOutputReadLine();
            prCompile.WaitForExit();
            prCompile.Close();
            prCompile.Dispose();

            CleanUpFolder(false, file);

            if (File.Exists(Config.Instance.WorkingDir + file.Remove(file.LastIndexOf('.')) + ".dll") || File.Exists(Config.Instance.WorkingDir + file.Remove(file.LastIndexOf('.')) + ".exe"))
            {
                dllExisting = true;
                linkFailure = false;
            }
            else
            {
                if (File.Exists(Config.Instance.WorkingDir + file.Remove(file.LastIndexOf('.')) + ".obj"))
                {
                    linkFailure = true;
                }
                dllExisting = false;
            }



        }
        catch (ThreadAbortException)
        {
            if (prCompile != null)
            {
                // On Error kill process
                prCompile.Kill();
                prCompile.Dispose();
            }
        }
        catch (Win32Exception)
        {
        }
        catch (Exception)
        {
            dllExisting = false;
        }

        while (true)
        {
            try
            {
                if (Directory.Exists(Dir.FullName))
                {
                    Directory.Delete(Dir.FullName, true);
                    break;
                }
                else
                {
                    break;
                }
            }
            catch
            {
            }
        }


    }
private void CopySource(string Destination)
{
    filesToCopy = new StringCollection();
    foreach (string strFile in Directory.GetFiles(c.WorkingDir))
    {
        string tmpStrFile = strFile.ToLower();

        foreach (string Extension in c.Extensions)
        {
            if (tmpStrFile.Contains(Extension))
            {
                filesToCopy.Add(tmpStrFile);
            }
        }
    }

    if (filesToCopy.Count > 0)
    {
        foreach (string strFile in filesToCopy)
        {
            File.Copy(strFile, Destination + strFile.Remove(0, strFile.LastIndexOf("\\")));
        }
    }
}

private void CleanUpFolder(bool PreCleanup, string Filename)
{
    //Copy all files from compilationfolder to working directory
    if (!PreCleanup)
    {
        foreach (string strFile in Directory.GetFiles(Dir.FullName, Filename.Remove(Filename.LastIndexOf(".") + 1) + "*"))
        {
            FileInfo fileToMove = new FileInfo(strFile);

            if (fileToMove.Name.ToLower().Contains(Filename.Remove(Filename.LastIndexOf("."))))
            {
                File.Copy(strFile, c.WorkingDir + fileToMove.Name, true);
            }
        }
    }

    //Delete useless files
    foreach (string filename in Directory.GetFiles(Config.Instance.WorkingDir, Filename.Remove(Filename.LastIndexOf("."))+".*"))
    {
        bool foundExt = c.Extensions.Contains(filename.Remove(0, filename.LastIndexOf(".") + 1));
        if (PreCleanup)
        {
            // Only delete files, which are not won't be compiled
            if(!foundExt)
            {
                File.Delete(filename);
            }
        }
        else
        {
            if (!Config.Instance.SaveLspFile && filename.Contains(".lsp"))
            {
                File.Delete(filename);
            }

            if (!Config.Instance.SaveLstFile && filename.Contains(".lst"))
            {
                File.Delete(filename);
            }
        }
    }
}

public void Dispose()
{
    Dispose(true);
    GC.SuppressFinalize(this);
}

protected virtual void Dispose(bool disposing)
{
    if (!disposed)
    {
        if (disposing)
        {
            Dir = null;
        }
        disposed = true;
    }
}

~CobolCompiler()
{
    Dispose (false);
}

I just tried it with sleeping two seconds after every compiling process. But this don't change anything.

While compilation progress the CPU is at 100%. The application is collecting 270 MB RAM. At start it's only 35MB.

Don't be affraid, i have to copy all sources to temp folders, because the compiler can't compile multiple files at same time in same working directory.

Edit: I already fixed the problem of no threads but still having items.

In ProcessSingle I add the item i tried to compile again, when it was not linked to an dll.

So I started with 14000 items and added items again (if they failed to link) to this concurrentBag while processing the Parallel.ForEach. So I end on 14000 runs of ForEach and have xxx modules which have to be compiled again. :-(

I didn't see that. The run of prReschk without WaitForExit is intended. Because of checking Ressources for more than 14000 items takes a long time and should not obstruct a new compilation.

But the problem of less threads at the end of the ConcurrentBag still exists :( But it is only notices, when it's a large amount of cycles.

Answer 1

If CopySource throws then you have an unreleased lock lockingObj and no further progress can be made. use lock (lockingObj) which makes use of a finally block to release the lock.

Answer 2

The Parallel.ForEach method will use the .Net ThreadPool to allocate a thread. The actual number of threads that will run in parallel will be governed by the ThreadPool depending on the the load of the system CPUs. So, you may have specified MaxDegreeOfParallelism but this is only the maximum, the ThreadPool may decide to allocate fewer threads than this maximum.

Based on the evidence you've given in your question, it sounds to me like the compilation process is using up system resources and not cleaning-up afterwards. This would explain why 140 compilations ends up in a gradual decline in the number of threads being allocated - the ThreadPool is not allocating new threads because it thinks that the CPU is heaviliy loaded.

I would look more closely at how the compilation process is terminated. Does the ProcessSingle method return before the compilation has fully completed? Is there a memory leak in the compilation process?

As an experiment, I would be interested to know if it behaved differently if you added the following line after calling ProcessSingle:

 System.Threading.Thread.Sleep(2000);

This will pause the thread for two seconds before passing control back to the ThreadPool to allocate the next task. If it improves your application's behaviour then it strongly suggests my theory is right.