CODEWITHSUNDEEP
vb.netvisual-studio-2010visual-studio-2012memory-leaks
Fariz Luqman
Fariz Luqman

Reputation: 914

How to avoid "Out Of Memory" exception when reading large files using File.ReadAllText(x)

This is my code to search for a string for all files and folders in the drive "G:\" that contains the string "hello":

Dim path = "g:\"
Dim fileList As New List(Of String)

 GetAllAccessibleFiles(path, fileList)

 'Convert List<T> to string array if you want
  Dim files As String() = fileList.ToArray

  For Each s As String In fileList
      Dim text As String = File.ReadAllText(s)
      Dim index As Integer = text.IndexOf("hello")
      If index >= 0 Then
           MsgBox("FOUND!")
           ' String is in file, starting at character "index"
      End If
  Next

This code will also results in memory leak/out of memory exception (as I read file as big as 5GB!). Perhaps it will bring the whole file to the RAM, then went for the string check.

Dim text As String = File.ReadAllText("C:\Users\Farizluqman\mybigmovie.mp4") 
    ' this file sized as big as 5GB!
    Dim index As Integer = text.IndexOf("hello")
    If index >= 0 Then
        MsgBox("FOUND!")
        ' String is in file, starting at character "index"
    End If

But, the problem is: This code is really DANGEROUS, that may lead to memory leak or using 100% of the RAM. The question is, is there any way or workaround for the code above? Maybe chunking or reading part of the file and then dispose to avoid memory leak/out of memory? Or is there any way to minimize the memory usage when using the code? As I felt responsible for other's computer stability. Please Help :)

Upvotes: 0

Views: 8021

Answers (1)

user2480047
user2480047

Reputation:

You should use System.IO.StreamReader, which reads line by line instead all the lines at the same time (here you have a similar post in C#); I personally never use ReadAll*** unless under very specific conditions. Sample adaptation of your code:

Dim index As Integer = -1
Dim lineCount As Integer = -1
Using reader As System.IO.StreamReader = New System.IO.StreamReader("C:\Users\Farizluqman\mybigmovie.mp4")
    Dim line As String
    line = reader.ReadLine
    If (line IsNot Nothing AndAlso line.Contains("hello")) Then
        index = line.IndexOf("hello")
    Else
        If (line IsNot Nothing) Then lineCount = line.Length
        Do While (Not line Is Nothing)
            line = reader.ReadLine
            If (line IsNot Nothing) Then
                lineCount = lineCount + line.Length
                If (line.Contains("hello")) Then
                    index = lineCount - line.Length + line.IndexOf("hello")
                    Exit Do
                End If
            End If
        Loop
    End If
End Using

If index >= 0 Then
    MsgBox("FOUND!")
    ' String is in file, starting at character "index"
End If

Upvotes: 3

Related Questions