.NET FileSystemWatcher goes into infinite loop when moving file

Question

I have a issue with the FileSystemWatcher. I'm using it in a windows service to monitor certain folders and when a file is copied, it proccesses that file using a SSIS package. Everything works fine, but every now and then, the FileWatcher picks up the same file and fires the Created event multiple times in a infinate loop. The code below works as follow:

Firstly, this method is called by the windows service and creates a watcher :

    private void CreateFileWatcherEvent(SSISPackageSetting packageSettings)
    {   
      // Create a new FileSystemWatcher and set its properties.
      FileSystemWatcher watcher = new FileSystemWatcher();

      watcher.IncludeSubdirectories = false;

      watcher.Path = packageSettings.FileWatchPath;

      /* Watch for changes in LastAccess and LastWrite times, and
        the renaming of files or directories. */

      watcher.NotifyFilter = NotifyFilters.LastAccess | NotifyFilters.LastWrite
        | NotifyFilters.FileName | NotifyFilters.DirectoryName | NotifyFilters.Size;

      //Watch for all files
      watcher.Filter = "*.*";

      watcher.Created += (s, e) => FileCreated(e, packageSettings);

      // Begin watching.
      watcher.EnableRaisingEvents = true;
    }

Next up, The Watcher.Created event looks something like this:

    private void FileCreated(FileSystemEventArgs e, SSISPackageSetting packageSettings)
    {
        //Bunch of other code not important to the issue

        ProcessFile(packageSettings, e.FullPath, fileExtension);
    }

The ProcessFile method looks something like this:

    private void ProcessFile(SSISPackageSetting packageSetting,string Filename,string fileExtension)
    {
        //COMPLETE A BUNCH OF SSIS TASKS TO PROCESS THE FILE


        //NOW WE NEED TO CREATE THE OUTPUT FILE SO THAT SSIS CAN WRITE TO IT      
        string errorOutPutfileName = packageSetting.ImportFailurePath + @"\FailedRows" + System.DateTime.Now.ToFileTime() + packageSetting.ErrorRowsFileExtension;
        File.Create(errorOutPutfileName).Close();

        MoveFileToSuccessPath(Filename, packageSetting);              
    }

Lastly, the MoveFile Method looks like this:

    private void MoveFileToSuccessPath(string filename, SSISPackageSetting ssisPackage)
    {
        try 
        {
            string newFilename = MakeFilenameUnique(filename);
            System.IO.File.Move(filename, ssisPackage.ArchivePath.EndsWith("\\") ? ssisPackage.ArchivePath + newFilename : ssisPackage.ArchivePath + "\\" + newFilename);

        }
        catch (Exception ex)
        {
            SaveToApplicationLog(string.Format
                ("Error ocurred while moving a file to the success path. Filename {0}. Archive Path {1}. Error {2}", filename, ssisPackage.ArchivePath,ex.ToString()), EventLogEntryType.Error);
        }            
    }

So somewhere in there, we go into a infinite loop and the FileWatcher keeps on picking up the same file. Anyone have any idea? This happens randomly and intermittently.

Answer 1

When using the FileSystemWatcher I tend to use a dictionary to add the files to when the notification event fires. I then have a separate thread using a timer which picks files up from this collection when they are more than a few seconds old, somewhere around 5 seconds.

If my processing is also likely to change the last access time and I watch that too then I also implement a checksum which I keep in a dictionary along with the filename and last processed time for every file and use that to suppress it firing multiple times in a row. You don't have to use an expensive one to calculate, I have used md5 and even crc32 - you are only trying to prevent multiple notifications.

EDIT

This example code is very situation specific and makes lots of assumptions you may need to change. It doesn't list all your code, just somethind like the bits you need to add:

// So, first thing to do is add a dictionary to store file info:

internal class FileWatchInfo
{
    public DateTime LatestTime { get; set; }
    public bool IsProcessing { get; set; }
    public string FullName { get; set; }
    public string Checksum { get; set; }
}
SortedDictionary<string, FileWatchInfo> fileInfos = new SortedDictionary<string, FileWatchInfo>();
private readonly object SyncRoot = new object();

//  Now, when you set up the watcher, also set up a [`Timer`][1] to monitor that dictionary.

CreateFileWatcherEvent(new SSISPackageSetting{ FileWatchPath = "H:\\test"});
int processFilesInMilliseconds = 5000;
Timer timer = new Timer(ProcessFiles, null, processFilesInMilliseconds, processFilesInMilliseconds);

// In FileCreated, don't process the file but add it to a list

private void FileCreated(FileSystemEventArgs e) {
    var finf = new FileInfo(e.FullPath);
    DateTime latest = finf.LastAccessTimeUtc > finf.LastWriteTimeUtc
        ? finf.LastAccessTimeUtc : finf.LastWriteTimeUtc;
    latest = latest > finf.CreationTimeUtc ? latest : finf.CreationTimeUtc;
    //  Beware of issues if other code sets the file times to crazy times in the past/future

    lock (SyncRoot) {
        //  You need to work out what to do if you actually need to add this file again (i.e. someone
        //  has edited it in the 5 seconds since it was created, and the time it took you to process it)
        if (!this.fileInfos.ContainsKey(e.FullPath)) {
            FileWatchInfo info = new FileWatchInfo {
                FullName = e.FullPath,
                LatestTime = latest,
                IsProcessing = false, Processed = false,
                Checksum = null
            };
            this.fileInfos.Add(e.FullPath, info);
        }
    }
}

And finally, here is the process method as it now is

    private void ProcessFiles(object state) {
        FileWatchInfo toProcess = null;
        List<string> toRemove = new List<string>();
        lock (this.SyncRoot) {
            foreach (var info in this.fileInfos) {
                //  You may want to sort your list by latest to avoid files being left in the queue for a long time
                if (info.Value.Checksum == null) {
                    //  If this fires the watcher, it doesn't matter, but beware of big files,
                    //     which may mean you need to move this outside the lock
                    string md5Value;
                    using (var md5 = MD5.Create()) {
                        using (var stream = File.OpenRead(info.Value.FullName)) {
                            info.Value.Checksum =
                                BitConverter.ToString(md5.ComputeHash(stream)).Replace("-", "").ToLower();
                        }
                    }
                }
                //  Data store (myFileInfoStore) is code I haven't included - use a Dictionary which you remove files from
                //   after a few minutes, or a permanent database to store file checksums
                if ((info.Value.Processed && info.Value.ProcessedTime.AddSeconds(5) < DateTime.UtcNow) 
                   || myFileInfoStore.GetFileInfo(info.Value.FullName).Checksum == info.Value.Checksum) {
                    toRemove.Add(info.Key);
                }
                else if (!info.Value.Processed && !info.Value.IsProcessing 
                    && info.Value.LatestTime.AddSeconds(5) < DateTime.UtcNow) {
                    info.Value.IsProcessing = true;
                    toProcess = info.Value;
                    //  This processes one file at a time, you could equally add a bunch to a list for parallel processing
                    break;
                }
            }
            foreach (var filePath in toRemove) {
                this.fileInfos.Remove(filePath);
            }
        }
        if (toProcess != null)
        {
            ProcessFile(packageSettings, toProcess.FullName, new FileInfo(toProcess.FullName).Extension); 
        }
    }

Finally, ProcessFile needs to process your file, then once completed go inside a lock, mark the info in the fileInfos dictionary as Processed, set the ProcessedTime, and then exit the lock and move the file. You will also want to update the checksum if it changes after an acceptable amount of time has passed.

It is very hard to provide a complete sample as I know nothing about your situation, but this is the general pattern I use. You will need to consider file rates, how frequently they are updated etc. You can probably bring down the time intervals to sub second instead of 5 seconds and still be ok.