Multithreading design-pattern

Question

While keeping in mind that:

• I am using a blocking queue that waits for ever until something is added to it
• I might get a FileSystemWatcher event twice

The updated code:

 {
        FileProcessingManager processingManager = new FileProcessingManager();
        processingManager.RegisterProcessor(new ExcelFileProcessor());
        processingManager.RegisterProcessor(new PdfFileProcessor());
        processingManager.Completed += new ProcessingCompletedHandler(ProcessingCompletedHandler);
        processingManager.Completed += new ProcessingCompletedHandler(LogFileStatus);
        while (true)
        {
            try
            {
                var jobData = (JobData)fileMonitor.FileQueue.Dequeue();
                if (jobData == null)
                    break;

                _pool.WaitOne();
                Application.Log(String.Format("{0}:{1}", DateTime.Now.ToString(CultureInfo.InvariantCulture), "Thread launched"));
                Task.Factory.StartNew(() => processingManager.Process(jobData));
            }
            catch (Exception e)
            {
                Application.Log(String.Format("{0}:{1}", DateTime.Now.ToString(CultureInfo.InvariantCulture), e.Message));
            }
        }
    }

What are are you suggestions on making the code multi-threaded while taking into consideration the possibility that two identical string paths may be added into the blocking queue? I have left the possibility that this might happen and in this case.. the file would be processed twice, the thing is that sometimes I get it twice, sometimes not, it is really awkward, if you have suggestions on this, please tell.

The null checking is for exiting the loop, I intentionally add a null from outside the threaded loop to determine it to stop.

Answer 1

For multi-threading this... I would probably add a "Completed" event to your FileProcessingManager and register for it. One argument of that event will be the "bool" return value you currently have. Then in that event handler, I would do the checking of the bool and re-queueing of the file. Note that you will have to keep a reference to the FileMonitorManager. So, I would have this ThreadProc method be in a class where you keep the FileMonitorManager and FileProcessingManager instances in a property.

To deduplicate, in ThreadProc, I would create a List outside of the while loop. Then inside the while loop, before you process a file, lock that list, check to see if the string is already in there, if not, add the string to the list and process the file, if it is, then skip processing.

Obviously, this is based on little information surrounding your method but my 2 cents anyway.

Rough code, from Notepad:

private static FileMonitorManager fileMon = null;
private static FileProcessingManager processingManager = new FileProcessingManager();

private static void ThreadProc(object param)
{
    processingManager.RegisterProcessor(new ExcelFileProcessor());
    processingManager.RegisterProcessor(new PdfFileProcessor());
    processingManager.Completed += ProcessingCompletedHandler;
    var procList = new List<string>();

    while (true)
    {
        try
        {
            var path = (string)fileMon.FileQueue.Dequeue();
            if (path == null)
                break;

            bool processThis = false;
            lock(procList)
            {
                if(!procList.Contains(path))
                {
                    processThis = true;
                    procList.Add(path);
                }
            }
            if(processThis)
            {
                Thread t = new Thread (new ParameterizedThreadStart(processingManager.Process));
                t.Start (path);
            }
        }
        catch (System.Exception e)
        {
            Console.WriteLine(e.Message);
        }
    }
}

private static void ProcessingCompletedHandler(bool status, string path)
{
    if (!status)
    {
        fileMon.FileQueue.Enqueue(path);
        Console.WriteLine("\n\nError on file: " + path);
    }
    else
        Console.WriteLine("\n\nSucces on file: " + path);
}