Making Reading of TextFile More efficient (Larger File Type)

Question

The following are my codes but it cant handle more than 500 lines at one time.

It needs to add a , to the end of the line and at the same time detect. What i'm currently doing is separating them into 2 different textbox then save the one which i need by copy pasting but the app seems to hang if the file is too big.

Can someone help me with making it more efficient. Would really appreciate it.

 private void button1_Click_1(object sender, EventArgs e)
    {
        if (openFileDialog1.ShowDialog() == DialogResult.Cancel)
            return;

        System.IO.StreamReader Reader = new System.IO.StreamReader(openFileDialog1.FileName);

        //Create a filestream

        FileStream fStr;

        try
        {
            //Set filestream to the result of the pick of the user

            fStr = new FileStream(openFileDialog1.FileName, FileMode.Open, FileAccess.Read);


            //Create a streamreader, sr, to read the file

            StreamReader sr = new StreamReader(fStr);

            //While the end of the file has not been reached...

            while (sr.Peek() >= 0)
            {
                //Create a 'line' that contains the current line of the textfile

                string line = sr.ReadLine().ToLower();

                if (line.Contains("staff"))
                {

                    line += ","; //Add a , to the end of the line**Important**
                    textBox1.Text += line + Environment.NewLine;
                    releventcount += 1;
                }
                else
                {
                    line += ","; //Add a , to the end of the line**Important**
                    textBox2.Text += line + Environment.NewLine;
                    irreleventcount += 1;
                }

                label1.Text = "Relevent: ";
                label2.Text = "Irrelevant: ";
            }
            //Close the file so other modules can access it
            sr.Close();
            //If something goes wrong, tell the user
        }
        catch (Exception)
        {

            MessageBox.Show("Error opening file", "Check the CODE ! ~.~");
        }


    }

Answer 1

I'm not sure what it is you're eventually trying to accomplish here. There are several more succinct ways to do what your current code is doing, but they won't significantly improve the speed of reading.

The bottleneck in your code is that you're appending strings. Using a StringBuilder is good advice, but you can do better than that by creating a List<string> and then calling string.Join at the end. For example:

if (openFileDialog1.ShowDialog() == DialogResult.Cancel)
    return;
List<string> staff = new List<string>();
List<string> other = new List<string>();

foreach (var line in File.ReadLines(openFileDialog1.FileName))
{
    line = line.ToLower();
    if (line.Contains("staff"))
    {
        staff.Add(line);
    }
    else
    {
        other.Add(line);
    }
}

relevantcount = staff.Count;
irrelevantCount = other.Count;

textBox1.Text = string.Join(","+Environment.NewLine, staff);
textBox2.Text = string.Join("."+Environment.NewLine, other);

Also, you say that your code can only handle 500 lines at a time. Is there something in your user interface that prevents it from handling more? Certainly, there's nothing in the code you showed that has such a low limit.

Answer 2

It is much, much more efficient to use StringBuilders to gather the text for the textboxes than to continuously append text.

Also, you should wrap your various streams in using blocks.

Here is a rewrite that should be much more efficient:

    private void button1_Click_1(object sender, EventArgs e)
    {
        if (openFileDialog1.ShowDialog() == DialogResult.Cancel)
            return;

        try
        {
            //Set filestream to the result of the pick of the user
            using (var fStr = new FileStream(openFileDialog1.FileName, FileMode.Open, FileAccess.Read))
            {
                //Create a streamreader, sr, to read the file
                using (var sr = new StreamReader(fStr))
                {
                    var sbTextBox1 = new System.Text.StringBuilder(10000);
                    var sbTextBox2 = new System.Text.StringBuilder(10000);

                    //While the end of the file has not been reached...
                    while (sr.Peek() >= 0)
                    {
                        //Create a 'line' that contains the current line of the textfile
                        string line = sr.ReadLine().ToLower();

                        if (line.Contains("staff"))
                        {
                            //Add a , to the end of the line**Important**
                            sbTextBox1.Append(line).Append(",").AppendLine();
                            releventcount += 1;
                        }
                        else
                        {
                            //Add a , to the end of the line**Important**
                            sbTextBox2.Append(line).Append(",").AppendLine();
                            irreleventcount += 1;
                        }
                    }

                    textBox1.Text = sbTextBox1.ToString();
                    textBox2.Text = sbTextBox2.ToString();

                    label1.Text = "Relevent: ";
                    label2.Text = "Irrelevant: ";

                    //Close the file so other modules can access it
                    sr.Close();
                }
            }
        }
        catch (Exception)
        {
            MessageBox.Show("Error opening file", "Check the CODE ! ~.~");
        }
    }

Answer 3

Reading files line by line is very slow. You can make this code much faster by reading a large block of data (or even the entire file if it's not too enormous). For example, use a File.ReadAllLines to read the entire file as separate lines, or use a FileStream and Read() into a buffer, and find the individual lines for yourself by looking for newline (\n, \r) characters.

To export the data, don't copy and paste it fom a text box - Write the results to one or two new files, and then just open them.

Answer 4

500 lines is nothing.

Try File.ReadAllLines and File.WriteAllLines.

Then you can do your work on an array of strings in memory and avoid the iterative IO.