CODEWITHSUNDEEP
c#multithreadingparallel-processingtask.net-4.5
zchpit
zchpit

Reputation: 3121

How to multithread long going task in C#

I have a problem. I have application, that generate pdf with reporting services. It works fine, but it need about 20-60 sec. to generate one pdf. This time is result of bug inside the reporting services. When I need one pdf by web, or 300 in one night it's ok, but now I need to make, that generate 7278 files. In one thread it will take about 4 days (about 96h). I need to make it works in one night (8-12h). I thinking about multithreding.

About code. Long running task (generate pdf) is generated in special web service, that is execiuted by console application. Console app, first take list of clients (clientsId) and raport endDate.

Code:

        DateTime batchDate = new DateTime(year, month, 1).GetEndOfMonthDate();
        int[] clientsId = GetClientsIds(batchDate);

        SaveLogFile(batchDate);
        LogProcessingStart(clientsId);

        int successedCount = 0;
        int processingCount = 1;

        CreateReportsDirectoryIfNecessary(ReportDir);
        int i = 0;

        while (i < clientsId.Length)
        {
            try
            {
                logger.Trace(string.Format("Przetwarzanie klient_id = {0}. {1}/{2}", clientsId[i], processingCount, clientsId.Length));
                ExecuteRequest(batchDate, clientsId[i]);
                logger.Trace(string.Format("Koniec przetwarzania klient_id = {0}", clientsId[i]));
                successedCount++;
            }
            catch (SoapException ex)
            {
                logger.Trace(string.Format("Błąd przetwarzania klient id = {0}.", clientsId[i]));
                logger.Trace(string.Format("Szczegóły : {0}", ex.Actor));
            }
            catch (Exception ex)
            {
                logger.Trace("Błąd " + ex.Message);
            }

            i++;
            processingCount++;
            Thread.Sleep(TaskDelaySeconds * 1000);
        }


protected override void ExecuteRequest(DateTime batchDate, int clientId)
{
    bool success = false;

    for (int i = 0; i < 10 && success == false; i++)
    {
        try
        {
            var result = gateway.GetPlatformReport(clientId, batchDate);
            string reportPath = ReportDir + "/" + result.ReportName + ".pdf";
            File.WriteAllBytes(reportPath, result.Report);
            success = true;
        }
        catch (Exception ex)
        {
            string origAssemblyLocation = Assembly.GetExecutingAssembly().CodeBase;
            logger.Trace("Błąd generowania raportu. Szczegóły " + ex.Message);
            logger.Trace(string.Format("Próbuje jeszcze raz.. (Retry = {0})", i + 1));
        }
    }
}

Long running task (generate pdf) is generated in ExecuteRequest(batchDate, clientsId[i]) that execute "gateway.GetPlatformReport(clientId, batchDate)" (what execute web service and take 20-60 sec., per client request). Both, the console app and web service is on the same machine.

At this moment, I have 7278 clients, and whole process take 4 days.

Upvotes: 1

Views: 762

Answers (3)

zmbq
zmbq

Reputation: 39023

Using Parallel.For in the client will not help you unless the web service can handle multiple requests concurrently. Make sure the web service runs multiple requests on multiple threads, and then you can use Parallel.For in your client as the others suggested.

That said, I would look into the PDF generation code. 60 seconds per file is quite a lot, perhaps you can speed it up.

UPDATE: ANOTHER OPTION If your Webservice already handles concurrent calls, you don't need to change your client application at all - just run multiple instances of the console application, and give each instance a different set of items.

Upvotes: 2

Sebastian Negraszus
Sebastian Negraszus

Reputation: 12215

Take a look at Parallel.For. This construct is similar to a for loop, but automatically handles partitioning of the workload and running the code on several threads.

Parallel.For(0, clientsId.Length, i => ExecuteRequest(batchDate, clientsId[i]));

Upvotes: 1

Samon
Samon

Reputation: 101

You could try replacing the while loop with a concurrent for loop.

Parallel.For(0, clientsId.Length, i =>
{
    // While body.
});

For more details see: https://msdn.microsoft.com/en-us/library/Ff963552.aspx and https://msdn.microsoft.com/en-us/library/system.threading.tasks.parallel.for%28v=vs.110%29.aspx

Upvotes: 2

Related Questions