How can I run two tasks concurrently?

Question

I have this for/next loop where I download a file and then process and load its contnets into a data base:

 For Each f As WinSCP.RemoteFileInfo In remotefilesinf
      If DownloadFile(FTPSession, CacheDirPath, "/mnt/usb", f) Then
           LoadDB(System.IO.Path.Combine(CacheDirPath, f.Name))
      Else
           MsgBox("Download failed.")
      End If
Next

In order to speed things up, how can I do the DB loading while the next file is downloading? I cannot do the DBLoad until each file download is complete and I can only do one DBLoad task at a time due to locking of the database.

I tried using a background worker for the LoadDB task but the RunWorkerCompleted event will not fire while the UI thread is busy with the download so i do not know when I can do the next DBload (DB not locked).

Any advice appreciated.

Answer 1

Here is another try since the requirement for the question have changed:

Public Class Form1

    Shared rnd As New Random

    Private download_que As New Queue(Of String)
    Private process_que As New Queue(Of String)
    Private download_thread As Thread
    Private process_thread As Thread

    Private Sub Form1_Load(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles MyBase.Load

    End Sub

    Private Sub Button1_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Button1.Click
        download_que.Enqueue("File 1.db")
        download_que.Enqueue("File 2.db")
        download_que.Enqueue("File 3.db")
        download_que.Enqueue("File 4.db")
        download_que.Enqueue("File 5.db")
        download_que.Enqueue("File 6.db")

        If download_thread Is Nothing then
            download_thread = New Thread(AddressOf DownloadFiles)
            download_thread.Start()
        End If
    End Sub

    Private AppendTextCaller As New Action(Of TextBox, String)(AddressOf AppendText)

    Public Sub AppendText(ByVal control As TextBox, ByVal text As String)
        control.AppendText(text)            
    End Sub

    Public Sub DownloadFiles()
        Dim file As String
        While download_que.Count > 0
            SyncLock download_que
                file = download_que.Dequeue()
            End SyncLock
            Dim path As String = Download(file)
            SyncLock process_que
                process_que.Enqueue(path)
            End SyncLock

            If process_thread Is Nothing Then
                process_thread = New Thread(AddressOf ProcessFiles)
                process_thread.Start()
            End If
        End While
        download_thread = Nothing
    End Sub

    Public Sub ProcessFiles()
        Dim path As String, ok As Boolean
        ok = True
        While process_que.Count > 0 And ok
            SyncLock process_que
                path = process_que.Dequeue()
            End SyncLock
            ok = LoadDB(path)
        End While
        process_thread = Nothing
    End Sub

    Public Function Download(ByVal filename As String) As String
        Dim sw = Stopwatch.StartNew()        
        Me.Invoke(AppendTextCaller, TextBox1, filename)
        Thread.Sleep(1500 + 500*rnd.Next(15))        
        Dim message As String = String.Format(" ({0:F1} sec)", sw.ElapsedMilliseconds / 1000)
        Me.Invoke(AppendTextCaller, TextBox1, message)
        Me.Invoke(AppendTextCaller, TextBox1, Environment.NewLine)
        Return IO.Path.Combine(IO.Path.GetTempPath(), filename)
    End Function

    Public Function LoadDB(ByVal path As String) As Boolean
        Dim sw = Stopwatch.StartNew()
        Dim filename = IO.Path.GetFileName(path)
        Me.Invoke(AppendTextCaller, TextBox2, filename)
        Thread.Sleep(800 + 500*rnd.Next(6))

        Dim message As String = String.Format(" ({0:F1} sec)", sw.ElapsedMilliseconds / 1000)
        Me.Invoke(AppendTextCaller, TextBox2, message)
        Me.Invoke(AppendTextCaller, TextBox2, Environment.NewLine)
        Return True
    End Function

End Class

Answer 2

I think this is what you want:

Public Function DownLoadFile(ByVal f As String) As String
    Trace.WriteLine("Start Downloading " & f)
    Dim x As Integer = ProgressBar1.Value
    Threading.Thread.Sleep(2000)        
    Me.Invoke(SetProgressCaller, x + 25)
    Trace.WriteLine("Done Downloading " & f)
    Return IO.Path.Combine(IO.Path.GetTempPath(), f)
End Function

Public Sub LoadDB(ByVal f As String)
    Trace.WriteLine("Start Loading " & f)
    Dim x As Integer = ProgressBar1.Value
    Threading.Thread.Sleep(1000)
    Me.Invoke(SetProgressCaller, x + 25)
    Trace.WriteLine("Done Loading " & f)
End Sub

Private Sub Button1_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Button1.Click
    ProgressBar1.Value = 0
    Dim f_path_1 = DownLoadFile("File 1")
    Dim t1 As New Threading.Thread(AddressOf LoadDB)
    t1.Start(f_path_1)
    ProgressBar1.Value = 50
    Dim f_path_2 = DownLoadFile("File 2")
    Dim t2 As New Threading.Thread(AddressOf LoadDB)
    t2.Start(f_path_2)        
End Sub

' Can be called using Form.Invoke() from any thread
Private SetProgressCaller As New Action(Of Integer)(AddressOf SetProgress)

' Set progress bar in main thread
Public Sub SetProgress(ByVal pct As Integer)
    ProgressBar1.Value = pct
End Sub

with the results:

Start Downloading File 1
Done Downloading File 1
Start Downloading File 2
Start Loading C:\Users\#####\AppData\Local\Temp\File 1
Done Loading C:\Users\#####\AppData\Local\Temp\File 1
Done Downloading File 2
Start Loading C:\Users\#####\AppData\Local\Temp\File 2
Done Loading C:\Users\#####\AppData\Local\Temp\File 2

which translates to

• Downloading of file 1 (takes 2 sec)
• Loading file 1 into DB (takes 1 sec) AND
• Downloading of file 2 (takes 2 sec)
• Loading of file 2 into DB (takes 1 sec)

Answer 3

You can run the DBLoad on a thread and set a ManualResetEvent to stop the execution before lauching the new DBLoad thread until the other ona finished.

Dim locker as New ManualResetEvent(True)

The locker acts like a traffic light, stoping the execution and waiting when is marked and going throught when otherwise.

Block the locker anywhere with:

locker.Reset()

Unblock the locker anywhere with:

locker.Set()

Set a stoping spot:

locker.WaitOne()

To see full capabilities see MSDN.

Answer 4

What about using two backgroundworkers? Use one to download files, the other one to stuff them into the db. If a download completes, append the file to a list and each time a db update finishes, look at that list from bgw2...