CODEWITHSUNDEEP
c#file-io
bLAZ
bLAZ

Reputation: 1751

Read large txt file and use it in chunks

I'm newbie in C# but I've made some research for reading/writing large txt files, the biggest might be 8GB but if it is too much I will consider split it maybe to 1GB. It must be fast up to 30MBytes/s. I've found three approaches: for sequential operation FileStream or StreamReader/StreamWriter, for random access MemoryMappedFiles. Now I'd like first to read file. Here is an example of code that works:

 FileStream fileStream = new FileStream(@"C:\Users\Guest4\Desktop\data.txt", FileMode.Open, FileAccess.Read);
try
{
    int length = (int)fileStream.Length;  // get file length
    buffer = new byte[length];            // create buffer
    int count;                            // actual number of bytes read
    sum = 0;                          // total number of bytes read

    // read until Read method returns 0 (end of the stream has been reached)
    while ((count = fileStream.Read(buffer, sum, length - sum)) > 0)
        sum += count;  // sum is a buffer offset for next reading
}
finally
{
    fileStream.Close();
}

Do you think is it good way to that fast big files reading?

After reading I need to resend that file. It must be in 16384 bytes chunks. Every chunk will be sent until all the data will be transmitted. And that chunks have to be string type. Could you suggest me how to do it? Split and convert to string. I suppose the best way is to send that string chunk not after reading all file, but if at least that 16384 bytes is read.

Upvotes: 0

Views: 4848

Answers (1)

bLAZ
bLAZ

Reputation: 1751

I've found something like this:

            FileStream FS = new FileStream(@"C:\Users\Guest4\Desktop\data.txt", FileMode.Open, FileAccess.ReadWrite);
            int FSBytes = (int) FS.Length;
            int ChunkSize = 1<<14;    // it's 16384
            byte[] B = new byte[ChunkSize];
            int Pos;

            for (Pos = 0; Pos < (FSBytes - ChunkSize); Pos += ChunkSize)
            {
                FS.Read(B,0 , ChunkSize);
                // do some operation on one chunk

            }

            B = new byte[FSBytes - Pos];
            FS.Read(B,0, FSBytes - Pos);
            // here the last operation procedure on the last chunk
            FS.Close(); FS.Dispose();

It seems to work. I hope only that this part: FS.Read(B,0 , ChunkSize); will be realy fast. If someone has anything to suggest, please do not hesitate.

Upvotes: 3

Related Questions