CODEWITHSUNDEEP
vb.net
RDM
RDM

Reputation: 11

Loading Large file into String Variable VB.NET

Question, if anyone could help please: The file I am reading from inPath is very large 300MB to 1 GB +. I need to load the file into the variable wholeFile as shown in the below program. Approximately 200 MB files works fine but larger files bomb out (Out of Memory Exception Error). The purpose is once file is loaded into the variable, I would need to run RegEx and pick certain section of the file and save somewhere else. Thanks once again for your kind attention.

Dim inPath As String = "C:\temp\300MB-File.txt" 
Dim outPath As String = "C:\temp\myFileNew2.txt"

        Dim wholeFile as String = ""

        Using sw As StreamWriter = File.CreateText(outPath)
            For Each oneLine As String In File.ReadLines(inPath)
                sw.WriteLine(oneLine)

                wholeFile = wholeFile & vbCrLf & oneLine

            Next 
    End Using

Upvotes: 0

Views: 2468

Answers (1)

jmcilhinney
jmcilhinney

Reputation: 54417

The way you're doing that is abominable. Why would you read a file line by line if your purpose is to store the entire contents in a single variable? Why wouldn't you load the whole file in one go?

Dim fileContents = File.ReadAllText(filePath)

That may still have memory issues with large files but the way you're doing will use exponentially more memory. Each time you do that concatenation to the String, you create a new String object and copy the previous contents into it along with the new text. That means that, for a file with N lines, you are going to create N Strings. The first will contain the first line, then the second will contain the first two lines, then the third will contain the first three lines, etc, etc.

If you really want to read the file line by line then you could use a StringBuilder, which avoids so much memory reallocation. Even better would be to get the size of the file first and then create the StringBuilder with the appropriate capacity from the get go, so no reallocation would be needed at all.

When you get right down to it though, files of that size are going to be an issue no matter what. You will either need to ensure that enough memory is allocated to your app to handle it or else you'll have to break the file up into chunks and process each chunk separately. If your regex won't match very large portions of the file then you can simply make each chunk overlap by a line or two and then handle the special cases where you get duplicate matches in the overlapping section.

Upvotes: 9

Related Questions