CODEWITHSUNDEEP
c#fileiofilestream
Juan Martinez
Juan Martinez

Reputation: 111

C# System.IO.FileStream performance hangs badly (10+ seconds)

I'm running into an issue wherein I am getting significant (10+ second) delays when performing file write operations. It seems only to happen once, and always happens during the 2nd (or sometimes 3rd?) call to the WriteToFile() function.

I've written out 3 different 'WriteToFile' functions to show some of the variations I've tried thus far + shown additional lines in 'OpenFileIfNecessary' that I've tried.

The code never throws an error, and the offsets/counts are all valid. Once the delays occur a single time, there seem to be no further delays.

This has been a pain in my side for 2+ days and I'm definitely at that point where I'm in need of a 2nd opinion.

    private void WriteToFile(byte[] data, long offset, int count)
    {
        lock (this.monitor)
        {
            this.OpenFileIfNecessary();
            this.fileStream.Seek(offset, SeekOrigin.Begin);  // <- Takes 10+ seconds for THIS line to execute
            this.fileStream.Write(data, 0, count);
        }
    }

    private void WriteToFile2(byte[] data, long offset, int count)
    {
        lock (this.monitor)
        {
            this.OpenFileIfNecessary();
            this.fileStream.Position = offset;    // <- Takes 10+ seconds for THIS line to execute 
            this.fileStream.Write(data, 0, count);
        }
    }

    private void WriteToFile3(byte[] data, long offset, int count)
    {
        lock (this.monitor)
        {
            var fileName = this.file.FullName;
            using (Stream fileStream = new FileStream(fileName, FileMode.OpenOrCreate))
            {
                fileStream.Position = offset;            //(instant execution of this line)
                fileStream.Write(data, 0, count);
                //Getting from HERE ->
            }
            //To HERE <- takes 10+ seconds
        }
    }

    private System.IO.FileStream fileStream = null;
    private System.IO.FileInfo file;  //value set during construction

    private void OpenFileIfNecessary()
    {
        lock (this.monitor) {
            if (this.fileStream == null) {
                //The following 3 lines all result in the same behavior described in this post

                //this.fileStream = this.file.Open(FileMode.OpenOrCreate, FileAccess.ReadWrite, FileShare.ReadWrite);
                //this.fileStream = this.file.Open(FileMode.OpenOrCreate, FileAccess.Write, FileShare.Write);
                //this.fileStream = this.file.OpenWrite();
                this.fileStream = this.file.Open(FileMode.OpenOrCreate);                    
            }               
        }
    }

Upvotes: 0

Views: 1241

Answers (2)

Tanveer Badar
Tanveer Badar

Reputation: 5523

That is happening because when you set a file position to some large value, underlying storage system has to zero out the contents of allocated blocks. I do not believe BCL will let you bypass that but there is actual a way in Win32 to skip that functionality which requires running program to have administrator privileges (in a very imprecise manner).

Search for SetFileValidData() documentation.

Upvotes: 0

Juan Martinez
Juan Martinez

Reputation: 111

Found the issue. It's worth mentioning that we had previously been testing with smaller (<1GB files) until late last week. With that in mind:

We write to the file at different positions, that is, we don't simply start at position 0 and go to the end. What that means (especially for larger files) is that every time we first go to a position that is deep into the file, there is apparently a wait period for the newly extended size to be allocated.

The way FileStream obfuscates a lot of the under-the-hood stuff made it a little difficult to find the pattern, and once we did some deeper profiling and discovered smaller delays with smaller files (never noticed the delays before) it became clear what was happening.

The plan forward is to do some multithreading to allow for the space for the file to be allocated fully before writing to disk; we can buffer in memory during that wait period.

Example code for preallocating the entire file:

fileStream.Seek(size - 1, SeekOrigin.Begin);
fileStream.WriteByte(0);
fileStream.Flush();

Upvotes: 2

Related Questions