CODEWITHSUNDEEP
c#streamreader
jLynx
jLynx

Reputation: 1149

StreamReader very slow for big files

I am wanting to read in a file which in this case is 3mb doing this takes around 50-60 seconds, which seems very slow. Does anyone know how to make this faster?

string text = null;
using (StreamReader sr = new StreamReader(file, Encoding.Default))
{
    string line;
    while ((line = sr.ReadLine()) != null)
    {
        text += (line);
        backgroundWorker1.ReportProgress(text.Length);
    }
}

I also need to use a background worker so I can report the percentage that has been loaded (for files around 500mb to 1gb)

Upvotes: 1

Views: 4534

Answers (3)

TomTom
TomTom

Reputation: 62093

StreamReader very slow for big files C#

No, it is not. The time you spend is not spent in the stream reader.

text += (line);

This line is creating a new string. For this is has to allocate new memory. For a large file this is a significant amount of garbage created. And the longer it gets, the more copy operations you do.

If that is what you use it for

backgroundWorker1.ReportProgress(text.Length);

It is also useless. You can also have a new variable

int textLength = 0

that you then set

textLength += line.Length

without all the text manipulation.

Performance problem? Never assume, always use a profiler.

Some background math, btw:

(for files around 500mb to 1gb)

That means that once you load 500mb of data your code is making a 500mb (if the file was Unicode) to 1gb (string double the size of an ascii file) copy operation per line.

You may want to look up the memory speed of your computer. Depending on server etc. you may be limited to 50gb per second (high end X99 - the newer DDR 4 memories are faster, but workstations generally have a lot less channels so are slower again) and a copy counts double (read and write). Which means that you really start running into "copying the string is overloading them memory bus" scenarios.

Upvotes: 3

RB.
RB.

Reputation: 37172

Use a StringBuilder to create your line - it's much more performant than string concatenation.

using System.Text;

//...

StringBuilder text = new StringBuilder();
using (StreamReader sr = new StreamReader(file, Encoding.Default))
{
    string line;
    while ((line = sr.ReadLine()) != null)
    {
        text.Append(line);
        backgroundWorker1.ReportProgress(text.Length);
    }
}

// ...
// Do something with the file you have read in.
Console.WriteLine(text.ToString());

Upvotes: 7

Tony Chu
Tony Chu

Reputation: 1

You can use this line:

string text = System.IO.File.ReadAllText(file);

Upvotes: 0

Related Questions