reading and writing multiple files at the same time and performing same tasks on them

Question

I am a beginner to programming. I wrote a code in C# to open a single file (that has 4 columns of data) and extract the fourth column into a list. Then did some basic work on the data to extract the mean, minimum and maximum values of the data set. Then, the results was written to dedicated files for the mean, minimum and maximum values.

Now I want to repeat the same tests but for a multiple sets of files - each with over 100,000 lines of data. I want to enable the program to read a multiple set of files in the same folder and then do the same calculations for each file and compile all the results for mean, minimum and maximum values into separate folders, as before.

The code for the single file is as follows;

    private void button1_Click_1(object sender, EventArgs e)
    {
        string text = "";
        DialogResult result = openFileDialog1.ShowDialog(); // Show the dialog.
        // create a list to insert the data into
        List<float> noise = new List<float>();
        int count = 0;
        float sum = 0;
        float mean = 0; 
        float max = 0;
        float min = 100;

        TextWriter tw = new StreamWriter("c:/Users/a3708906/Documents/Filereader - 13062012/Filereader/date.txt");
        if (result == DialogResult.OK) // Test result.
        {
            string file = openFileDialog1.FileName;
            FileInfo src = new FileInfo(file);
            TextReader reader = src.OpenText();
            text = reader.ReadLine();

            // while the text being read in from reader.Readline() is not null

            while (text != null)
            {
                text = reader.ReadLine();
                if (text != null)
                {
                    string[] words = text.Split(',');
                    noise.Add(Convert.ToSingle(words[3]));

                    // write text to a file
                    tw.WriteLine(text);
                    //foreach (string word in words)
                    //{
                    //    tw.WriteLine(word);
                    //}
                }

            }
        }

        tw.Close();

        TextWriter tw1 = new StreamWriter("c:/Users/a3708906/Documents/Filereader - 13062012/Filereader/noise.txt");
        foreach (float ns in noise)
        {
            tw1.WriteLine(Convert.ToString(ns));
            count++;
            sum += ns;
            mean = sum/count;

            float min1 = 0; 

            if (ns > max)
                max = ns;
            else if (ns < max)
                min1 = ns;

            if (min1 < min && min1 >0)
                min = min1;
            else
                min = min;

        }

        tw1.Close();

        TextWriter tw2 = new StreamWriter("c:/Users/a3708906/Documents/Filereader - 13062012/Filereader/summarymeans.txt");
        tw2.WriteLine("Mean Noise");
        tw2.WriteLine("==========");
        tw2.WriteLine("mote_noise 2: {0}", Convert.ToString(mean));
        tw2.Close();

        TextWriter tw3 = new StreamWriter("c:/Users/a3708906/Documents/Filereader - 13062012/Filereader/summarymaximums.txt");
        tw3.WriteLine("Maximum Noise");
        tw3.WriteLine("=============");    
        tw3.WriteLine("mote_noise 2: {0}", Convert.ToString(max));
        tw3.Close();

        TextWriter tw4 = new StreamWriter("c:/Users/a3708906/Documents/Filereader - 13062012/Filereader/summaryminimums.txt");
        tw4.WriteLine("Minimum Noise");
        tw4.WriteLine("=============");
        tw4.WriteLine("mote_noise 2: {0}", Convert.ToString(min));
        tw4.Close();
    }

I will be grateful if someone could help to translate this code for working with multiple files. Thank you.

Answer 1

Wrap your logic for processing a single file into a single Action or a void-returning function, then enumerate the files, switch them to ParallelEnumerable and call Parallel.ForAll

For example, if you made an Action or function named DoStuff(string filename) which will do the process for a single file, you can then call it with :

Directory.EnumerateFiles(dialog.SelectedPath).AsParallel().ForAll(doStuff);

Answer 2

Your current code will work if you simply use Directory.GetFiles() properly. The easiest way to do it would be to have three inputs; one to get the Directory, and a second to get the file extension (if wanted), and a checkbox to ask whether or not you want to recursively search the folders or not.

Then instead of

        string file = openFileDialog1.FileName;

you would instead have something like

        //ensure the default fileExtensionDropdown.SelectedValue is "*"
        string[] filePaths;
        if(chkRecursiveSearch.IsChecked == true)
           filePaths = Directory.GetFiles(dlgFolderBrowser.SelectedPath, @"*"+ddlFileExtension.SelectedValue, SearchOption.AllDirectories);
        else
           filePaths = Directory.GetFiles(dlgFolderBrowser.SelectedPath, @"*"+ddlFileExtension.SelectedValue);

Then you can use:

        for(string path in filePaths){ // do things }

to handle each file path the way you are right now.

Please note the code I've put here is definitely not as idiomatic and tidy as it could be, but since you said you were a beginner I decided to be a bit more clear. If requested I'll put up a more idiomatic take on things, though if we do that we should probably clean up your initial code a bit as well.