c# multithreading issue waiting for threads

Question

After reviewing many posts on this forum and others about c# multithreading, I am still confused and having problems handling the issue at hand.

I want to create numThreads number of threads to execute a function for each integer in documentIds. documentIds is a List<int> with Count = 100 and I want to call RestitchWorker.Restitch on each of these elements in documentIds.

What I have currently is below but I am confused on how to keep 5 threads going and cycling through the list of 100 documentIds... all help is appreciated.

for (int i = 0; i < documentIds.Count; i++)
{
    if (threadCount < numThreads)
    {
        var Worker = new RestitchWorker(documentIds.ElementAt(i));
        Thread t_restitchWorker = new Thread(() => Worker.Restitch());
        threadCount++;
        t_restitchWorker.Start();
        t_restitchWorker.Join();
     }
}

Answer 1

There are lots of mechanisms for multi-threading an application. You could use async tasks, task parallel library (TPL), threadpools, semaphores, etc. The mechanism I've taken to using when I need a high degree of control over how many threads I generate at once is to use a SemaphoreSlim, set the max number of threads, and then use Wait to wait for a new thread to become available. I'll put my data (eg. list of document Ids) into a ConcurrentQueue so that I can safely queue and dequeue information my threads need to perform work without having to worry (too much) about thread safety. Async tasks can be used to actually start performing the work.

static class Program
{
    private const int _maxWorkers = 10;
    private static readonly SemaphoreSlim _threadManager = new SemaphoreSlim(_maxWorkers);

    private void DoStuff()
    {
        var queue = new ConcurrentQueue<int>(SomeClass.GetDocumentIds());
        var operations = new List<Task>();

        while (!queue.IsEmpty)
        {
            int documentId;

            if (queue.TryDequeue(out documentId))
            {
                _threadManager.Wait(); // this will block after max number of threads is met
                try
                {
                    operations.Add(GetDocument(documentId);
                }
                catch (Exception)
                {
                    // ignored
                }
            }
        }

        Task.WaitAll(operations.ToArray()); // finally, this waits till the last remaining tasks are complete
    }

    private static async Task GetDocument(int documentId)
    {
        await Task.Yield();

            try
            {
                GoGetDocument(documentId);
            }
            catch (Exception)
            {
                // ignored
            }
            finally
            {
                _threadManager.Release(); // release the semaphore when the task completes
            }

    }

Answer 2

This problem is really better suited to Tasks not Threads. If for your problem creating threads explicitly isn't necessary; then thread pooling can be more efficient due to the fact that thread management can be expensive.

Tasks suit problems in which 100 long jobs need to be achieved loosely in parallel because they will pool threads rather than creating 100 explicit threads, and have an easy to use API that supports waiting on multiple tasks.

List of Tasks

You can do this by creating a list of tasks and calling System.Threading.Tasks.Task.WaitAll:

var tasks = new List<Task>();

for (int i = 0; i < n; ++i)
{
    tasks.Add(Task.Run(action: SomeLongTask));
}

Task.WaitAll(tasks);

Parallel.For

However a better way to do this since it's in a for loop and you need to pass an integer is to use System.Threading.Tasks.Parallel.For:

Parallel.For(0, n, (i) =>
{
   SomeLongTask(i); 
});