Executing multiple threads

Question

I am developing Windows Form C# program which reads Excel data from shared drive every 20 minutes (I'm using "Timer") - function "inserting". I want to read multiple Excel files at once because of the performance. For that reason I'm using threads.

Each thread is calling a function (LoadExcelData) which reads data from Excel to ArrayList. I want to know when all threads are finished (when all excel files were loaded to ArrayList) in order to insert this ArrayList to internal database.

I tried with thread[i].Join() but this freezes GUI. I also do not know what would happen if I have 100+ files and for this reason 100+ threads. Would that cause memory exception or some other exception?

        //Execute every 20 minutes  (Timer). Do not Execute in case previouse run is not finished
        void inserting(List<String> excels){

        int numOfThreads=excels.length;
        Thread[] threads = new Thread[numOfThreads];
        for (int index = 0; index < numOfThreads; index++)
        {
            int i = index;
            threads[index] = new Thread(() =>
                {
                    LoadExcelData(excels[i].File_name); //function loads excel data to global array "Weather" which is used later on
                });
        }

       for (int i = 0; i < threads.Length; i++)
        {
            threads[i].Start(); //start thread
        }

        for (int i = 0; i < threads.Length; i++)
        {
            //   threads[i].Join(); //this freezes GUI!
        }

       InsertToDB(object of ArrayList<ClassName>); //insert data which was read from Excels

       isRunning=false;//Data was successefully inserted to DB 
     }

I want to run this every 20 minutes. I'm using Timer:

    timer = new System.Windows.Forms.Timer();
    timer.Tick += new EventHandler(timerEventHanlder);
    timer.Interval = 20 * 60000; // in miliseconds
    timer.Start(); 

private void timerEventHanlder(object sender, EventArgs e)
{
   List<String> excels = getExcels();
    if (!isRunning){ //in case previous timer even is not finished wait another 20 minutes...
        isRunning=true; //flag to true
        inserting(excels);
       }
}

Is there any better wait to solve above problem?

Answer 1

The UI thread is freezing because you're using a System.Windows.Forms.Timer which fires the timer ticked event on the UI thread; this is useful in that you don't have to Invoke anything on the tick event. Calling Join blocks the calling thread and in your case this is the UI thread.

To avoid this (and since you're not needing to Invoke any UI elements), you can change your System.Windows.Forms.Timer to a System.Timers.Timer, which runs in a thread separate from the UI thread. If you switch to a System.Timers.Timer, you'll need to change some of the syntax in your code (e.g. the Tick event is the Elapsed event instead, etc.).

There's also the System.Thread.Timer and the System.Web.UI.Timer, additionally, you could also spawn a second thread from within the timer tick event to avoid it waiting on the threads within the UI thread, example:

private void timerEventHanlder(object sender, EventArgs e)
{
    (new System.Threading.Thread(() => {
        List<String> excels = getExcels();
        if (!isRunning){ //in case previous timer even is not finished wait another 20 minutes...
            isRunning=true; //flag to true
            inserting(excels);
        }
    })).Start();
}

Starting a new thread avoids changing any of your current code and allows you to change it back if you do ever need to invoke anything in the UI.

Answering you're other question though:

I also do not know what would happen if I have 100+ files and for this reason 100+ threads. Would that cause memory exception or some other exception?

Spawning 100+ threads won't cause any exceptions unless your code has a specific exception (like a null delegate passed as the ThreadStart), or if the OS can't create a thread, which if the OS can't create a thread you have bigger problems. It is possible that memory exhaustion could happen since the Thread is a managed object and thus takes up memory (along with an ArrayList, but the amount of memory for 100+ threads (even 1000+) is negligible on any system that is capable of running the .NET framework (even on most embedded systems), so the number of threads won't necessarily be an issue.

Looking at your code, you might want to consider instead of spawning 100+ threads, utilizing the System.Threading.ThreadPool and a System.Threading.CountDownEvent, example:

CountdownEvent Countdown;

void LoadExcelData(object data)
{
    // loads excel data to global array "Weather" which is used later on
    Countdown.Signal();
}

//Execute every 20 minutes  (Timer). Do not Execute in case previouse run is not finished
void inserting(List<object> excels)
{
    Countdown = new CountdownEvent(excels.Count); 
    int i = 0;
    while (i < excels.Count) {
        ThreadPool.QueueUserWorkItem(LoadExcelData, excels[i++].File_name);
    }
    Countdown.Wait();

    InsertToDB(WeatherList); //insert data which was read from Excels
    isRunning = false; //Data was successefully inserted to DB 
}

This will utilize the system thread pool to execute your functions and allows .NET to handle the scheduling of the threads to avoid massive resource contention if the number of threads is a lot. You could use other methods to block, like a Mutex or Semaphore, but the CountDownEvent pretty much encapsulates what you'd need to do with other wait objects and joining on the threads from the thread pool.

To be honest though, since you're reading data from Excel files in multiple threads, unless each thread reads the entire contents of the file into RAM then executes the operations that way, you might not see a huge increase in performance. Multi-threaded applications that have heavy I/O usually don't see a huge performance increase unless said I/O is on performance minded equipment or the initial input of the entire file is read into RAM. Just a side note as you're multi-threading with files.

It should also be noted too that utilizing the System.Threading.ThreadPool is ideally for threads you expect to only run for a few seconds or so; if you anticipate that a thread could take longer, you should stick with spawning the threads as you have now. You can still use the CountDownEvent and you don't need an array of threads like you have (you could just just use the (new Thread(function)).Start() syntax).

Hope that can help

Answer 2

The parent thread is going to reach the for loop that joins all the worker threads and wait there until all the threads have finished (and can be joined). If the GUI is running in that same parent thread, execution is not going to return to the GUI until all threads have finished, which is going to be a long time as you've set up timers. Try running the GUI in a different thread.

Edit: Also on a side note, I'd set your timer lengths to something much shorter while you're debugging to see if it's actually waiting as you expect it to. Then once you have it functioning correctly you can set it back to 20 minutes.