Creating and Deleting files quickly causes an exception

Question

I've got some code running in a single threaded process targeting .Net 4.5.2. This code creates a file, does some stuff and then deletes the file. Most of the time this works just fine, however, there are occasions where an exception is thrown.

I've tried to simplify and recreate the problem below. As you can see from the screenshot, it manages to create and delete the file 240+ times, then fails with UnauthorizedAccessException. Sometimes the code below runs without error, sometimes it gets farther through the loop. So what is going on here?

In a production environment, files are written to and deleted from a network share, when using a network share an IOException is thrown as follows (so it breaks in a slightly different place, during the File.Delete rather the ):

System.IO.IOException: The process cannot access the file '\\network_location\sub_dir\TMP1.sql' because it is being used by another process.
   at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
   at System.IO.File.InternalDelete(String path, Boolean checkHost)

As I understand it, the using statements should call dispose, causing the filestreams to be flushed; so after the using statement I expect all the data to be written to disk.

How can I ensure this works consistently as I expect?

    static void Main(string[] args)
    {
        string networkFilePath = @"C:\Temp";
        string filename1 = "tmp1.sql";
        string filename2 = "tmp2.sql";

        string tempFilename1 = Path.Combine(networkFilePath, filename1);
        string tempFilename2 = Path.Combine(networkFilePath, filename2);

        int interations = 20000;

        Console.WriteLine("Started");
        try
        {
            for (int i = 0; i < interations; i++)
            {
                using (var s = new StreamWriter(tempFilename1))
                {
                    s.Write("aaa");
                }

                using (var s = new StreamWriter(tempFilename2))
                {
                    s.Write("aaa");
                }

                FileCompare(tempFilename1, tempFilename2);

                File.Delete(tempFilename1);
                File.Delete(tempFilename2);
            }
        }
        catch (IOException e)
        {
            Console.WriteLine(e.Message);
        }

        Console.WriteLine("Finished");
        Console.ReadLine();
    }

    public static bool FileCompare(string filename1, string filename2)
    {
        using (var fs1 = new FileStream(filename1, FileMode.Open))
        using (var fs2 = new FileStream(filename2, FileMode.Open))
        {
            return [... ommited for brevity, but just checks the content of file1 vs file2]
        }
    }

edit I'm aware I can use temporary files names to avoid the issue... the question is, how do I know for certain that an IO operation is complete, eg, that cached data is written to disk either locally or on the network?

edit I have amended my code to close each stream, but still get the following...

Answer 1

Sometimes nothing but either the OS or the GC are holding onto the file handle, both out of your control, and not worth the effort to monitor the I/O, ... Your objective seems to be reliable cleanup, not I/O monitoring.

If you wanted to take a threaded approach, you could queue these file paths to a spinwait thread for exception tolerant cleanup.

If you didn't want to add a thread, then the 'exit cleanup loop upon exception' could be placed at the bottom of your processing cycle.

Either way, you have answer your own appropriate approach for files that stubbornly refuse to be deleted. Consider the following pseudo-code approach of queuing the files to be cleaned up into a Dictionary that tracks the deletion retries to avoid an infinite loop:

// ... declarative section, downgrade to Dictionary class if thread-safe concurrency is not needed
ConcurrentDictionary<string, int> files_for_cleanup = new ConcurrentDictionary<string, int>();

// ... bottom of your processing loop
if (!files_for_cleanup.ContainsKey(current_temp_file))
{
    files_for_cleanup.TryAdd(current_temp_file, 0);
}

// ... switching to cleanup thread, if threaded, otherwise continue with the bottom of your processing loop
KeyValuePair<string, int> item = null;
try
{
  while(!files_for_cleanup.IsEmpty)
  {
    item = files_for_cleanup.First();
    if(File.Exists(item.Key)
    {
      File.Delete(item.Key);
    }
    files_for_cleanup.TryRemove(item.Key, item.Value);
  }
}
catch(Exception)
{
  if (item != null)
  {
    if (item.Value > 40 /* retry threshold */)
    {
      files_for_cleanup.TryRemove(item.Key, item.Value);
      // ... perform other unrecoverable actions, e.g. logging, delete on reboot, ...
    }

    else
    {
      files_for_cleanup.TryUpdate(item.Key, item.Value, item.Value + 1);
    }
  }
}

//  ... if in thread, spinwait, sleep, etc.  otherwise return to top of loop

Answer 2

Most often this happens when some other program opens the same file with permissive share mode which includes FileShare.Delete. This share allows another process to delete a file, but file will not really be deleted until all handles to it are closed (including handle obtained by process which opened it with FileShare.Delete). Examples of such programs might be search indexers and antivirus software - both might like to scan new file you created, and both don't want to lock a file, so will open it with very permissive share mode, including FileShare.Delete.

When you call File.Delete on a file which is already open with FileShare.Delete - it will return without any errors and file will be marked for deletion (and will be deleted after all handles are closed). Any subsequent attempt to access such "deleted" file will throw access denied exception you observe in your example.

You can easily reproduce that with the following code:

static void Main(string[] args) {
    var tmpFile = Path.GetTempFileName();
    // file is now created
    new Thread(() => {
        using (var fs = new FileStream(tmpFile, FileMode.Open, FileAccess.Read, FileShare.ReadWrite | FileShare.Delete)) {
            // open with permissive share mode, including FileShare.Delete
            // and hold that handle for 5 seconds
            Thread.Sleep(5000);
        }
    }).Start();
    // wait a bit for thread above to grab a handle
    Thread.Sleep(1000);
    // delete, will return fine without any errors            
    File.Delete(tmpFile);
    // attempt to create - will fail with access denied exception you have
    using (var s = new StreamWriter(tmpFile)) {
        s.Write("aaa");
    }
    Console.WriteLine("done");
    Console.ReadLine();
}

Because you have no idea how long another process would hold a handle to file you are going to delete - I think all you can do is just check if file still exists after deletion and if yes - wait a bit and check again:

File.Delete(tmpFile);
while (File.Exists(tmpFile)) {
    // obviously don't loop forever
    // check for some time then throw exception
    Thread.Sleep(100);
}
// in the example above it will loop for 4 seconds while
// handle is hold by another thread, then finally return
// and subsequent creation of new file will work fine without exceptions

Answer 3

As Drag and Drop has commented, add .Close(), it should really be called. Just to make sure the stream is closed.

If that doesn't do the trick, you could use random or increasing (1,2,3,...) file names, since they are temporary anyway. Queue these files and have a second thread try to delete them, until it succeeds.