CODEWITHSUNDEEP
multithreadingazureazure-functionsparallel.foreachazure-function-app
vikky
vikky

Reputation: 171

Multi-threading in Azure

I wanted to shift a windows service to azure by the means of running it through an Azure function.

The windows service runs multi-threading using Parallel.ForEach loop and just to test whether the multithreading works in Azure perfectly or not , I wrote the below code:

Azure Function :

[FunctionName("Function1")]
public static void Run([TimerTrigger("0 */5 * * * *")] TimerInfo myTimer, TraceWriter log)
{
    try
    {
        Process process = new Process();

        process.StartInfo.FileName = @"D:\home\site\wwwroot\Dependency\MasterVB\MasterVB.exe";
        process.StartInfo.Arguments = "";
        process.StartInfo.UseShellExecute = false;
        process.StartInfo.RedirectStandardOutput = true;
        process.StartInfo.RedirectStandardError = true;

        log.Info("Before Calling Start ");
        process.Start();
        log.Info("After Calling Start ");

        string output = process.StandardOutput.ReadToEnd();
        string err = process.StandardError.ReadToEnd();
        process.WaitForExit();
    }
    catch
    {
      //left out for brevity
    }
}

MasterVB.exe:

Imports System.Collections.Concurrent
Imports Microsoft.WindowsAzure.Storage
Imports Microsoft.WindowsAzure.Storage.Blob
    
Module Module1
    
    Public connectionString As String = "DefaultEndpointsProtocol=https;AccountName=r53eripcjroswtest;AccountKey=cktlWe/byeSOis94OOA0YFlHPRrAfmORuy9xRGScTrxwuliY4KmZfAif8aF4cgng2mnDLvJWIvxl0xEPMU6DKw==;"
    Public storageAccount As CloudStorageAccount = CloudStorageAccount.Parse(connectionString)
    Public serviceClient As CloudBlobClient = storageAccount.CreateCloudBlobClient()
    Public container As CloudBlobContainer = serviceClient.GetContainerReference("igb-test")
    Public blob As CloudAppendBlob = container.GetAppendBlobReference("DocumentMaster.txt")
    
    Sub Main()
    
        If Not blob.Exists Then
            blob.CreateOrReplace()
        End If
    
        Dim dealist As New List(Of String)
    
        dealist.Add("First")
        dealist.Add("Second")
        dealist.Add("Third")
        dealist.Add("Fourth")
        dealist.Add("Fifth")
        dealist.Add("Sixth")
        dealist.Add("Seventh")
        dealist.Add("Eighth")
        dealist.Add("Nine")
        dealist.Add("Tenth")
        Dim exceptions As New ConcurrentQueue(Of Exception)
        Dim opts As New ParallelOptions
        opts.MaxDegreeOfParallelism = -1
        Parallel.ForEach(dealist, opts, Sub(deal)
                                            If Len(deal) > 0 Then
                                                Try
                                                    Dim p As New Process()
                                                    p.StartInfo.FileName = "D:\home\site\wwwroot\Dependency\MasterVB\ChildVB\ChildVB.exe"
                                                    p.StartInfo.Arguments = deal
                                                    p.StartInfo.UseShellExecute = False
                                                    p.StartInfo.CreateNoWindow = True
                                                    p.Start()
                                                    p.WaitForExit()
                                                Catch ex As Exception
                                                    exceptions.Enqueue(ex)
                                                End Try
                                            End If
                                        End Sub)
    
        If exceptions.Count > 0 Then
            For Each ex In exceptions
                'Console.WriteLine(ex.Message)
                blob.AppendText(ex.Message + Environment.NewLine)
            Next
        End If
    
    End Sub
    
End Module

ChildVB.exe

Imports Microsoft.WindowsAzure.Storage
Imports Microsoft.WindowsAzure.Storage.Blob
Module Module1
    Public keycode As String = ""
    Public dealPass As String = ""
    Public runType As String = ""
    
    Public connectionString As String = "DefaultEndpointsProtocol=https;AccountName=r53eripcjroswtest;AccountKey=cktlWe/byeSOis94OOA0YFlHPRrAfmORuy9xRGScTrxwuliY4KmZfAif8aF4cgng2mnDLvJWIvxl0xEPMU6DKw==;"
    Public storageAccount As CloudStorageAccount = CloudStorageAccount.Parse(connectionString)
    Public serviceClient As CloudBlobClient = storageAccount.CreateCloudBlobClient()
    Public container As CloudBlobContainer = serviceClient.GetContainerReference("igb-test")
    Public blob As CloudAppendBlob = container.GetAppendBlobReference("DocumentChild.txt")
    
    Sub Main()
        Dim clArgs() As String = Environment.GetCommandLineArgs()
    
        If Not blob.Exists Then
            blob.CreateOrReplace()
        End If
            
        If clArgs.Count = 2 Then
    
            Dim args As String = clArgs(1)
            blob.AppendText(args + "----" + DateTime.Now.ToString("MM/dd/yyyy HH:mm:ss") + Environment.NewLine)                
    
        End If
    End Sub
    
End Module

In the DocumentChild.txt , i am getting the following output:

First----08/12/2020 15:15:03
Sixth----08/12/2020 15:15:04
Third----08/12/2020 15:15:05
Seventh----08/12/2020 15:15:06
Fourth----08/12/2020 15:15:07
Nine----08/12/2020 15:15:09
Tenth----08/12/2020 15:15:10
Fifth----08/12/2020 15:15:10

First----08/12/2020 15:20:17
Sixth----08/12/2020 15:20:17
Second----08/12/2020 15:20:18
Seventh----08/12/2020 15:20:19
Fifth----08/12/2020 15:20:21
Eighth----08/12/2020 15:20:21
Nine----08/12/2020 15:20:23
Tenth----08/12/2020 15:20:23
Fourth----08/12/2020 15:20:24

First----08/12/2020 15:25:02
Sixth----08/12/2020 15:25:03
Second----08/12/2020 15:25:04
Seventh----08/12/2020 15:25:04
Third----08/12/2020 15:25:05
Fifth----08/12/2020 15:25:06
Eighth----08/12/2020 15:25:07
Fourth----08/12/2020 15:25:07
Nine----08/12/2020 15:25:09
Tenth----08/12/2020 15:25:09

First----08/12/2020 15:30:01
Sixth----08/12/2020 15:30:01
Second----08/12/2020 15:30:03
Third----08/12/2020 15:30:03
Seventh----08/12/2020 15:30:04
Fifth----08/12/2020 15:30:05
Nine----08/12/2020 15:30:06
Fourth----08/12/2020 15:30:06
Eighth----08/12/2020 15:30:07
Tenth----08/12/2020 15:30:08

First----08/12/2020 15:35:02
Second----08/12/2020 15:35:04
Third----08/12/2020 15:35:04
Fifth----08/12/2020 15:35:06
Seventh----08/12/2020 15:35:06
Fourth----08/12/2020 15:35:07
Nine----08/12/2020 15:35:08
Eighth----08/12/2020 15:35:08
Tenth----08/12/2020 15:35:09

Above lines shows the output at every different azure function run after 5 mins for five different times. The output in the first and second instance, at 15:15 and 15:20 do not print all the ten strings but does so in the third and fourth instance but does not print all the strings in the fifth instance. In the actual code, the deallist mentioned in MasterVB will contain more than 4,000 strings to process in the same way,i.e., master calling child using mutlithreading.

My Questions are:

  1. Why is the multithreading in this case not consistent and prints all ten strings every time? My concern is if its inconsistent with just 10 strings , will it be able to process 4000 strings with
    more operations than just printing to a text file in the ChildVB module?
  2. Is there any good alternative to shift this windows service,which involves multi-threading, to azure.
  3. Is there a possibility this could be related to deployment plan.

Note:

1.I am using consumption plan to deploy but not facing any issues such as timeout which is there in this plan as a standard(10 mins)

  1. I have ran this for several different times , but the inconsistency persists everytime the azure function is run.
  2. There is no complilation or runtime error.

Upvotes: 0

Views: 2659

Answers (1)

PramodValavala
PramodValavala

Reputation: 6647

While it may not be possible to identify why all the runs did not complete without detailed logs, I can suggest a solution for your scenario (and many similar scenarios like this one).

Firstly, instead of a list of strings, you could insert a message into a queue (service bus or storage queue) for each string that you need to process. You could even have a HTTP Trigger function that inserts messages into the queue using the appropriate output binding (service bus or storage queue).

Then, depending on the queue services that you opted for, you will have a function triggered by its appropriate trigger (service bus or storage queue). This function will scale out depending on the number of messages if you are running it on the consumption or premium plan.

If you require a subset of messages to be processed in order, then you would have to go with service bus utilizing its message sessions. There is a configuration setting for the service bus trigger that you will have to set for it to accept message sessions.

Upvotes: 1

Related Questions