Are native OS operations optimized on a per-process basis while writing same file?

Question

I'm trying to understand some internal details of file writing operation done through managed code in C#. Let's say I write below piece of code to write some content into a log file:

using System.IO;
public void Log(string logMessage, LogLevel logLevel)
{
    if (Directory.Exists(LogFileDirectory))
    {
        using (StreamWriter streamWriter = new StreamWriter(LogFileDirectory + "log.txt",true))
        {
            streamWriter.WriteLine(DateTime.Now.ToString() + " - " + logMessage);                                    
        }
    }
}

To write a file on disk, I can list down few things which must be happening:

• Search for the file at the designated path
• Checking user permission access to the directory and path. Opening file handle to any file on disk require you to go through native OS calls which must be access controlled.
• Opening file handles
• Loading disk drivers (This probably happens at OS boot time itself but require confirmation on this also)
• Actual disk rotation
• hard disk drive head/spindle movement
• Creating I/O buffers etc.
• Pulling blocks of data from disk to RAM.
• <<I could be missing few more as I'm not very thorough with disk I/O at OS level>>

The thing or internal detail I'm trying to understand is that do the above steps get repeated for each new instance of StreamWriter class that I create in the application or there are few things which operating system (OS) or process can optimize or cache if every time it is same process who is asking to write something to the exactly same file?

Answer 1

As far as understand your question, it can be broken down into the following.

1. The internal detail I'm trying to understand is, do the above steps get repeated for each new instance of StreamWriter class that I create in the application

2. Are there things which the operating system (OS) or process can optimize or cache if every time it is same process who is asking to write something to the exactly same file

3. Your special bounty requirement "I also want to understand if OS applies some extra intelligence when the file read/write requests are coming from same process or different processes? Or does OS remains agnostic of the process who is requesting the read/write operation".

---

#To answer the first question

Disclaimer : The below is only relevant to the actual code you have written (as is). If it’s changed slightly a lot of the implementation details become irrelevant.

Actually little of what you describe above get repeated (or actually even done) every time you create a StreamWriter. However, things certainly do happen.

Let’s take a journey through .Net Source creating a StreamWriter the way you have.

Creating StreamWrtier

using (StreamWriter streamWriter = new StreamWriter(LogFileDirectory + "log.txt",true))

The call chain is as follows

1. public StreamWriter(String path, bool append)

Initializes a new instance of the StreamWriter class for the specified file by using the default encoding and buffer size. If the file exists, it can be either overwritten or appended to. If the file does not exist, this constructor creates a new file.

2. public StreamWriter(String path, bool append, Encoding encoding, int bufferSize)

3. internal StreamWriter(String path, bool append, Encoding encoding, int bufferSize, bool checkHost)

4. private static Stream CreateFile(String path, bool append, bool checkHost)

Which calls the following specifically with the FileMode.Append flag.

Opens the file if it exists and seeks to the end of the file, or creates a new file. This requires FileIOPermissionAccess.Append permission. FileMode.Append can be used only in conjunction with FileAccess.Write. Trying to seek to a position before the end of the file throws an IOException exception, and any attempt to read fails and throws a NotSupportedException exception.

5. internal FileStream(String path, FileMode mode, FileAccess access, FileShare share, int bufferSize, FileOptions options, String msgPath, bool bFromProxy, bool useLongPath, bool checkHost)

As you can see, and for anyone playing at home, all we done is create a file stream. From here we Marshal some Security Attributes then call:

6. private void Init(String path, FileMode mode, FileAccess access, int rights, bool useRights, FileShare share, int bufferSize, FileOptions options, Win32Native.SECURITY_ATTRIBUTES secAttrs, String msgPath, bool bFromProxy, bool useLongPath, bool checkHost)

There is a lot that goes on at this point; Checking permissions; Checking what the file type is; Checking handles. However, the crux of the story is:

7. internal static SafeFileHandle SafeCreateFile(String lpFileName, int dwDesiredAccess, System.IO.FileShare dwShareMode, SECURITY_ATTRIBUTES securityAttrs, System.IO.FileMode dwCreationDisposition, int dwFlagsAndAttributes, IntPtr hTemplateFile)

Resolving in a DllImport

[DllImport(KERNEL32, SetLastError=true, CharSet=CharSet.Auto, BestFitMapping=false)]
[ResourceExposure(ResourceScope.Machine)]
private static extern SafeFileHandle CreateFile(String lpFileName, int dwDesiredAccess, System.IO.FileShare dwShareMode, SECURITY_ATTRIBUTES securityAttrs, System.IO.FileMode dwCreationDisposition, int dwFlagsAndAttributes, IntPtr hTemplateFile);

This is the end of our .Net story ending in our beloved KERNEL32 CreateFile Function

Creates or opens a file or I/O device. The most commonly used I/O devices are as follows: file, file stream, directory, physical disk, volume, console buffer, tape drive, communications resource, mailslot, and pipe. The function returns a handle that can be used to access the file or device for various types of I/O depending on the file or device and the flags and attributes specified.

If you have ever used CreateFile you will know lots of information here about Flags, Caching and Buffering, and quite-frankly lots of stuff that is not even relevant to the File System. That's because this is up-there with one of the oldest User Mode API calls there is and it does all-sorts-of-things. Though, if you follow the .Net source (in this situation) it actually doesn’t use much of its extended nature.

The only major exception being the :

• FILE_APPEND_DATA

To write to the end of file, specify both the Offset and OffsetHigh members of the OVERLAPPED structure as 0xFFFFFFFF. This is functionally equivalent to previously calling the CreateFile function to open hFile using FILE_APPEND_DATA access.

• FILE_FLAG_OVERLAPPED flag which signals Async IO (which you did not set in this situation)

The file or device is being opened or created for asynchronous I/O.

When subsequent I/O operations are completed on this handle, the event specified in the OVERLAPPED structure will be set to the signaled state.

If this flag is specified, the file can be used for simultaneous read and write operations.

If this flag is not specified, then I/O operations are serialized, even if the calls to the read and write functions specify an OVERLAPPED structure.

Synchronous and Asynchronous I/O Handles

If a file or device is opened for synchronous I/O (that is, FILE_FLAG_OVERLAPPED is not specified), subsequent calls to functions such as WriteFile can block execution of the calling thread until one of the following events occurs:

• The I/O operation completes (in this example, a data write).
• An I/O error occurs. (For example, the pipe is closed from the other end.)
• An error was made in the call itself (for example, one or more parameters are not valid).
• Another thread in the process calls the CancelSynchronousIo function using the blocked thread's thread handle, which terminates I/O for that thread, failing the I/O operation.
• The blocked thread is terminated by the system; for example, the process itself is terminated, or another thread calls the TerminateThread function using the blocked thread's handle. (This is generally considered a last resort and not good application design.)

Synchronous and Asynchronous I/O

In some cases, this delay may be unacceptable to the application's design and purpose, so application designers should consider using asynchronous I/O with appropriate thread synchronization objects such as I/O completion ports. For more information about thread synchronization, see About Synchronization. A process opens a file for asynchronous I/O in its call to CreateFile by specifying the FILE_FLAG_OVERLAPPED flag in the dwFlagsAndAttributes parameter. If FILE_FLAG_OVERLAPPED is not specified, the file is opened for synchronous I/O. When the file has been opened for asynchronous I/O, a pointer to an OVERLAPPED structure is passed into the call to ReadFile and WriteFile. When performing synchronous I/O, this structure is not required in calls to ReadFile and WriteFile.

CreateFile provides for creating a file or device handle that is either synchronous or asynchronous. A synchronous handle behaves such that I/O function calls using that handle are blocked until they complete, while an asynchronous file handle makes it possible for the system to return immediately from I/O function calls, whether they completed the I/O operation or not. As stated previously, this synchronous versus asynchronous behavior is determined by specifying FILE_FLAG_OVERLAPPED within the dwFlagsAndAttributes parameter. There are several complexities and potential pitfalls when using asynchronous I/O; for more information see Synchronous and Asynchronous I/O,

I/O Completion Ports

I/O completion ports provide an efficient threading model for processing multiple asynchronous I/O requests on a multiprocessor system. When a process creates an I/O completion port, the system creates an associated queue object for requests whose sole purpose is to service these requests. Processes that handle many concurrent asynchronous I/O requests can do so more quickly and efficiently by using I/O completion ports in conjunction with a pre-allocated thread pool than by creating threads at the time they receive an I/O request.

How I/O Completion Ports Work

The CreateIoCompletionPort function creates an I/O completion port and associates one or more file handles with that port. When an asynchronous I/O operation on one of these file handles completes, an I/O completion packet is queued in first-in-first-out (FIFO) order to the associated I/O completion port. One powerful use for this mechanism is to combine the synchronization point for multiple file handles into a single object, although there are also other useful applications. Please note that while the packets are queued in FIFO order they may be dequeued in a different order.

Note : FileStream can use Completion Ports, by setting the useAsync true in one of the FileStream Overloads, however you did not

public FileStream(String path, FileMode mode, FileAccess access, FileShare share, int bufferSize, bool useAsync)

The actual write

You have chosen WriteLine() which is actually TextWriter method, however before we start lets just make a little note on FileStream. To get atomically-appended writes to shared log files working

• The FileStream internal buffer needs to be big enough to hold all of the data for a single write
• Data has to be written into the buffer from position 0 so that it fits
• Text written through the StreamWriter wrapping the FileStream has to be completely flushed before each write

Of these requirements, the first one (1) is the least pleasant to work around. The buffer size is fixed when a FileStream is created (the 4096 argument above), so the only way to atomically write a larger event is to close and reopen the file with a larger buffer.

Flushing the StreamWriter and FileStream between writes neatly takes care of requirements (2) and (3).

1. public virtual void WriteLine(String value)

2. public virtual void Write(char\[\] buffer, int index, int count)

Which surprising does this little gem :

for (int i = 0; i < count; i++) Write(buffer[index + i]);

3. public virtual void Write(char value)

When the the buffer is full it calls, a chain of flushes, this is kind of hard to follow but I'll try to simply

if (charPos == charLen) Flush(false, false);

Where charLen = DefaultBufferSize which was passed into one of the constructors by default where you created the StreamWriter and defined as follows :

internal const int DefaultBufferSize = 1024;   // char[]

4. private void Flush(bool flushStream, bool flushEncoder)

From here the 2 most important things are :

if (count > 0)
     stream.Write(byteBuffer, 0, count);

// By definition, calling Flush should flush the stream, but this is
// only necessary if we passed in true for flushStream.  The Web
// Services guys have some perf tests where flushing needlessly hurts.
if (flushStream)
     stream.Flush();

Note : You have to love MS source code comments, those kids are a total riot

The second Flush() (if you follow it through) will just end in the following anyway. Remembering our StreamWriter is backed by a FileStream class so we end up once again the FileStream class Write method

5. public override void Write(byte\[\] array, int offset, int count)

6. private unsafe void WriteCore(byte\[\] buffer, int offset, int count)

7. private unsafe int WriteFileNative(SafeFileHandle handle, byte\[\] bytes, int offset, int count, NativeOverlapped* overlapped, out int hr)

Resolving another DllImport

[DllImport(KERNEL32, SetLastError=true)]
[ResourceExposure(ResourceScope.None)]
internal static unsafe extern int WriteFile(SafeFileHandle handle, byte* bytes, int numBytesToWrite, out int numBytesWritten, IntPtr mustBeZero);

Then before you know it we are back in KERNEL32 again calling the WriteFile Function

Writes data to the specified file or input/output (I/O) device. This function is designed for both synchronous and asynchronous operation. For a similar function designed solely for asynchronous operation

Once again, there is a boat-load of options that deal with all sorts of situations. Yet once again, .Net (in this situation) doesn't tend to make use of much of it.

From here our story switches to the Windows file caching, which we have little control over in .Net, however you can use lots of options in the Raw Api Calls.

By default, Windows caches file data that is read from disks and written to disks. This implies that read operations read file data from an area in system memory known as the system file cache, rather than from the physical disk. Correspondingly, write operations write file data to the system file cache rather than to the disk, and this type of cache is referred to as a write-back cache. Caching is managed per file object.

So disregarding the actual Kernel Mode what have we actually done here in User Mode? Not much... When creating a Stream .Net has done a bunch of checks-and-balances to call a simple CreateFile Win32 Api Call, that in-turn it holds a handle. Sure there is a bunch of IL that gets called but at its basic-level its just using the Win32 API to create a file and hold a Handle (boiling off some of the .Net security and permission checking etc...)

Then what happens? Well, we deal with some encoding, write some bytes to memory, then at a predetermined buffer-size we write/flush it to Disk using FileWrite Win32 Api Call.

Has the Operating System had-to-do anything it wouldn’t have had-to-do for any other simple User Mode file create and write? Actually not really...

The only caveat begin, once again, sure .Net does its song-and-dance and if you really wanted to have Atomic Access to the File System from User Mode then consider calling these function yourself and/or use IO Completion ports. This way you get the advantage of async work and/or dodging .Net and being able access a bunch of extended parameters (though in most cases it wont make it any faster in a single application as the API default parameters already optimized for general purpose situations such as writes).

If you really have Processor Instruction OCD, then yes its easy to see its worth while either keeping the FileStream around and continuing to write to it in a append mode (taking precautions), or if you are at the Win32 Api level, creating the file and holding the handle for continuous writes and using the Asynchronous IO facilities.

But here is the clincher.. that’s all you can really do from User Mode as a User Mode programmer (in most cases).

---

#To answer the second question

As previously stated, there is not much you can do apart from; Roll your own API Calls; Install a fast HDD (that might in some circumstances use its own Driver). But regardless of this, the operating system already caches and optimizes regardless. If you want to tweak this, once again you need to go to the Windows API and/or use more advanced async IO feature

Lastly different Server Operating system versions, have advantages in memory and caching, though this all very well documented

---

#The answer to the third question

There is no preferential treatment given to the process writing to an open file in a multiple threaded application (.Net internal buffers aside) as opposed to multiple process writing to the same file I am aware of (however someone more experience maybe elaborate more). At User Mode everyone gets the same Apis to work with which go through same Filter Drivers and Caching Manager. You can read more about the Caching manager and improvements in operating systems here .

From MSDN

Caching occurs under the direction of the cache manager, which operates continuously while Windows is running. File data in the system file cache is written to the disk at intervals determined by the operating system, and the memory previously used by that file data is freed—this is referred to as flushing the cache. The policy of delaying the writing of the data to the file and holding it in the cache until the cache is flushed is called lazy writing, and it is triggered by the cache manager at a determinate time interval. The time at which a block of file data is flushed is partially based on the amount of time it has been stored in the cache and the amount of time since the data was last accessed in a read operation. This ensures that file data that is frequently read will stay accessible in the system file cache for the maximum amount of time.

The amount of I/O performance improvement that file data caching offers depends on the size of the file data block being read or written. When large blocks of file data are read and written, it is more likely that disk reads and writes will be necessary to finish the I/O operation. I/O performance will be increasingly impaired as more of this kind of I/O operation occurs.

---

#Supplemental

All the above is purely just academic now. We can dive deeper into the source code to the Windows API; We can follow it through to IO Completion ports and Kernel Mode; We can dissect the Drivers, but the answers to your questions aren't going to get any more clearer... Well apart from what we already know.

If you really want more knowledge on how the internals will react under pressure, the next step you need to take is to run your own Benchmarks to gain actual implementation and Empirical Evidence. You would do this by; Testing in both .Net and customized parameters at the Windows Api level with the same and different processors to figure out if overhead is relevant; Using different hardware; Playing around with different Operating Systems (like servers which are engineered with different fine grained control over caching); With different Drivers that using different physics pathways to your physical device.

---

#Summary

In short, apart from keeping the Stream alive (we already know Creating/Opening a file will incur a penalty); Using Async IO operations; Using the appropriate buffer sizes; Sending in extended parameters into Win32 Api's directly. There really isn't much more I can see apart from running your own performance testing on different Operating Systems and configurations and hardware, which will be able to give you more answers into what's more efficient and performant in your situation.

I hope this helps, or at least gives people a fun journey through the StreamWriter to the API (in those 2 simple calls).

Answer 2

I would have to start by asking why is this important, even if it isn't in native c# there wouldn't be much you could do to change this as the calls technically exist outside an editable section of code it's part of the system.dll which get interpreted by the os through a pipeline. How it's interpreted at the OS level matters very little, just know it works. How the OS handles it across all os when you get into xamarin mono takes over and sends a basic read or write command to the os for processing on its stack as it's free to do so (sometimes this may take a while depending on how busy things are in the mobile land). You can not speed this up or optimize this with ease (you would basically be building a new language and no one recommends taking that on.)

As ajay said above in the end you shouldn't really care because in the long run you can't really change the behavior of it.

you could read some material here for how I/O in general works and optimization attempts:

https://www.wilderssecurity.com/threads/trying-to-understand-i-o-nomenclature.286453/

https://learn.microsoft.com/en-us/dotnet/standard/io/

http://enterprisecraftsmanship.com/2014/12/13/io-threads-explained/

hope this helps you understand a little more and what you are seeking (I'm still not sure what you want to know or why. You might be better off hitting up the .net core team for more information)

Answer 3

The .NET classes are just tip of the iceberg, a very thin layer on top of native Win32 functions. There is a lot happening when you cross the managed layer.

A simplified view:

    .NET layer (managed)
-----------------------------
    Win32 layer (User Mode)
-----------------------------
    Drivers (Kernel Mode)

For actual disk I/O, you can consider the "Win32 layer" as a medium-thick layer. Actual disk I/O, checking if the file is present or not, access permissions etc. don't happen at this layer. Forget about physical disk movement. This layer, in coordination with the Kernel (which constitutes device drivers, file-system drivers, IO manager, and other system components) would check if handle is valid, if parameters passed are valid, if operation being performed is good (like "write" operation for file opened with "read-only" will be denied). If you see the code of .NET classes in a decompiler like dotPeek or any other similar tool, you'd see the managed code calling the Win32 APIs.

The actual disk I/O is performed by the Kernel mode components. This is the thickest layer, the core, the tentacle of Disk I/O (or Network I/O). Be it accessing the physical disk-drive, performing security checks, processing asynchronous IO, initiating APC/DPC, calling other drivers in the driver-stack (file-system driver, mini-filter driver, monitors etc.), ensuring that all files handles are closed when process exits (which aren't closed by application, explicitly). Anti-virus components would run at this level, and they would either record the file I/O operation, prevent the operation, or even modify the operation altogether. Most disk I/O, on a bare OS, would be performed by drivers provided by Microsoft. Anti-virus, specific device-driver (like for your favorite hard-disk), other monitoring drivers (like used by Process Monitor, or WireShark) and other drivers may be installed and would participate in disk I/O requests made by the user mode/.NET application.

Most of the drivers are loaded at boot time, or on demand. They are not loaded when file open/close/read/write calls are made.

Windows is a complex and big operating system. Many I/O elements are scattered in kernel world (largely) and in Win32 world (sub-system DLLs). You cannot say that "file-permission" is performed by only Kernel or User - it is an amalgamation of both. Caching, memory-manager, storage manager and many other "lower" User/Kernel components do it for the user application.

Different versions of Windows would do things differently.

You cannot say that Kernel IO would be fastest, and .NET IO would be slowest. It would be almost same. Though a trip to the Kernel (from User (.NET inclusive)) would cost some CPU cycles, and hence an application should ideally minimize the IO calls e.g. read 10K bytes, rather than 10 bytes 1000 times.

In the end, all I would say is - you shouldn't care!