Optimization of Listbox Population

Question

I'm having some trouble creating an optimized function to split strings within a text file, and populate my ListView accordingly. For the function below, it's taking 32 seconds to get through 40,000 lines.

The only goal is to take each line of a text file, delimited by a space, and input them into the corresponding column. Any suggestions on a faster way to sort through these lines? I did also try Regulator Expression for splitting, but that did not help.

This blog had some great benchmarking for what I'm trying to accomplish, but I'm having trouble adapting it to my function: http://cc.davelozinski.com/c-sharp/fastest-way-to-read-text-files

    Dim ColDate As String
    Dim Time As String
    Dim Action As String
    Dim Protocol As String
    Dim Source As String
    Dim Destination As String
    Dim SourcePort As String
    Dim DestPort As String

    Dim logFileStream As FileStream = New FileStream(My.Settings.LogLocation, FileMode.Open, FileAccess.Read, FileShare.ReadWrite)
    Dim reader As StreamReader = New StreamReader(logFileStream)
    Dim line As String

    Do
        line = reader.ReadLine
        Try
                'Make sure it's not a comment
                If line.StartsWith("#") Then
                    Exit Try
                End If

                Dim split As String() = line.Split(" ")
                ColDate = split(0)
                Time = split(1)
                Action = split(2)
                Protocol = split(3)
                Source = split(4)
                Destination = split(5)
                SourcePort = split(6)
                DestPort = split(7)

                frmMain.lstLogs.Items.Add(ColDate)
                frmMain.lstLogs.Items(frmMain.lstLogs.Items.Count - 1).SubItems.Add(Time)
                frmMain.lstLogs.Items(frmMain.lstLogs.Items.Count - 1).SubItems.Add(Action)
                frmMain.lstLogs.Items(frmMain.lstLogs.Items.Count - 1).SubItems.Add(Protocol)
                frmMain.lstLogs.Items(frmMain.lstLogs.Items.Count - 1).SubItems.Add(Source)
                frmMain.lstLogs.Items(frmMain.lstLogs.Items.Count - 1).SubItems.Add(Destination)
                frmMain.lstLogs.Items(frmMain.lstLogs.Items.Count - 1).SubItems.Add(SourcePort)
                frmMain.lstLogs.Items(frmMain.lstLogs.Items.Count - 1).SubItems.Add(DestPort)

            Catch ex As Exception
                'MsgBox(ex.ToString)
            End Try
        'End If
    Loop Until reader.EndOfStream
    reader.Close()

Answer 1

Exclude array from code

Dim split As String() = line.Split(" ")

Replace lines of code

With line
    ColDate = .Split(" ")(0)
    Time = .Split(" ")(1)
    Action = .Split(" ")(2)
    Protocol = .Split(" ")(3)
    Source = .Split(" ")(4)
    Destination = .Split(" ")(5)
    SourcePort = .Split(" ")(6)
    DestPort = .Split(" ")(7)
End Wiht

With frmMain.lstLogs
  .Items.Add(ColDate)
  .Items(frmMain.lstLogs.Items.Count - 1).SubItems.Add(Time)
  .Items(frmMain.lstLogs.Items.Count - 1).SubItems.Add(Action)
  .Items(frmMain.lstLogs.Items.Count - 1).SubItems.Add(Protocol)
  .Items(frmMain.lstLogs.Items.Count - 1).SubItems.Add(Source)
  .Items(frmMain.lstLogs.Items.Count - 1).SubItems.Add(Destination)
  .Items(frmMain.lstLogs.Items.Count - 1).SubItems.Add(SourcePort)
  .Items(frmMain.lstLogs.Items.Count - 1).SubItems.Add(DestPort)
 End Wiht

OR

With frmMain.lstLogs
  .Items.Add(line.Split(" ")(0))
  .Items(frmMain.lstLogs.Items.Count - 1).SubItems.Add(line.Split(" ")(1))
  .Items(frmMain.lstLogs.Items.Count - 1).SubItems.Add(line.Split(" ")(2))
  .Items(frmMain.lstLogs.Items.Count - 1).SubItems.Add(line.Split(" ")(3))
  .Items(frmMain.lstLogs.Items.Count - 1).SubItems.Add(line.Split(" ")(4))
  .Items(frmMain.lstLogs.Items.Count - 1).SubItems.Add(line.Split(" ")(5))
  .Items(frmMain.lstLogs.Items.Count - 1).SubItems.Add(line.Split(" ")(6))
  .Items(frmMain.lstLogs.Items.Count - 1).SubItems.Add(line.Split(" ")(7))
 End Wiht

Answer 2

Don't update a User Interface inside a loop without .BeginUpdate and .EndUpdate. I was working with a ListView with approximately 1500 rows and they appear instantaneously.

I created a list of ListViewItem s. Then looped through my data adding the details of each new ListViewItem and adding it to the list. After the loop a .ToArray allowed me to use the .AddRange method of the ListView. This is surrounded by begin and end update. No screen redraws speeds things up a great deal.

Private Sub OpCode()
    Dim lstListItems As New List(Of ListViewItem)
    Dim lines = File.ReadAllLines(My.Settings.LogLocation) 'Assume this is the path to the file you are reading
    For Each line In lines
        'Make sure it's not a comment
        If line.StartsWith("#") Then
            Continue For
        End If

        Dim split As String() = line.Split(" "c)

        Dim li As New ListViewItem
        li.Text = split(0) 'ColDate
        li.SubItems.Add(split(1)) 'Time
        li.SubItems.Add(split(2)) 'Action
        li.SubItems.Add(split(3)) 'Protocol
        li.SubItems.Add(split(4)) 'Source
        li.SubItems.Add(split(5)) 'Destination
        li.SubItems.Add(split(6)) 'SourcePort
        li.SubItems.Add(split(7)) 'DestPort

        lstListItems.Add(li)
    Next
    'Assume that lstLogs is the name of your ListView control
    lstLogs.BeginEdit
    lstLogs.Items.AddRange(lstListItems.ToArray)
    lstLogs.EndEdit
End Sub

Answer 3

Instead of using a StreamReader , I think you should read the text file then loop over the lines. Try this ::

Dim ColDate As String
Dim Time As String
Dim Action As String
Dim Protocol As String
Dim Source As String
Dim Destination As String
Dim SourcePort As String
Dim DestPort As String

Dim logFile As String() = IO.File.ReadAllLines(My.Settings.LogLocation)
For Each line in logFile
    Try
        'Make sure it's not a comment
        If line.StartsWith("#") Then
            Exit Try
        End If

        Dim split As String() = line.Split(" ")
        ColDate = split(0)
        Time = split(1)
        Action = split(2)
        Protocol = split(3)
        Source = split(4)
        Destination = split(5)
        SourcePort = split(6)
        DestPort = split(7)

        frmMain.lstLogs.Items.Add(ColDate)
        frmMain.lstLogs.Items(frmMain.lstLogs.Items.Count - 1).SubItems.Add(Time)
        frmMain.lstLogs.Items(frmMain.lstLogs.Items.Count - 1).SubItems.Add(Action)
        frmMain.lstLogs.Items(frmMain.lstLogs.Items.Count - 1).SubItems.Add(Protocol)
        frmMain.lstLogs.Items(frmMain.lstLogs.Items.Count - 1).SubItems.Add(Source)
        frmMain.lstLogs.Items(frmMain.lstLogs.Items.Count - 1).SubItems.Add(Destination)
        frmMain.lstLogs.Items(frmMain.lstLogs.Items.Count - 1).SubItems.Add(SourcePort)
        frmMain.lstLogs.Items(frmMain.lstLogs.Items.Count - 1).SubItems.Add(DestPort)

    Catch ex As Exception
        'MsgBox(ex.ToString)
    End Try
Next