FileSystemWatcher on Windows Service .NET 8 [IHostedService]

Question

Im trying to create a windows service application with .NET 8 using as a Hosted Service, the idea of my app is Watch for a list of folders. The problem is that I'm losing the event Created fired by the FileSystemWatcher. In context my app should works in this way:

1. A file is created inside watcher folders.
2. The file is validated by extension (.txt or .zip) and not empty content.
3. The file is uploaded to AWS Bucket.
4. The file is removed from the folder.

These is my happy path, when Im debugging works fine for some files, but if the upload of file is very heavy, then the app lose the events for the FileSystemWatcher.

I'm trying resolve this with Producer/Consumer pattern. I follow these articles:

https://medium.com/@dayanandthombare/background-services-in-net-core-c32f15e28678#:~:text=%E2%9C%85%20Solution%3A%20Background%20Service%20for%20Asynchronous%20File%20Processing

FileSystemWatcher losing files in its queue

But not working to me.

My code looks:

// Worker.cs
public class Worker : BackgroundService
{
private readonly ConcurrentQueue<FileTask> _fileTasks = new ConcurrentQueue<FileTask>();
private readonly SemaphoreSlim _signal = new SemaphoreSlim(0);

protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        try
        {
            _logger.LogInformation("Worker running at: {time}", DateTimeOffset.Now);
            _logger.LogInformation("Starting sync files.");

            _synchronizer.SetUp();

            while (!stoppingToken.IsCancellationRequested)
            {
                await _signal.WaitAsync(stoppingToken);

                if (_fileTasks.TryDequeue(out FileTask? task) && task is not null && task.Process is not null && task.FileRelation is not null)
                {
                    await task.Process(task.FileRelation, stoppingToken);
                }

                await Task.Delay(TimeSpan.FromSeconds(5), stoppingToken);
            }
        }
        catch (Exception exception)
        {
            _logger.LogError($"An unhandled error has occurred. Exception {exception}");
            throw;
        }
    }
}

// FileServiceSynchronizer.cs
public class FileServiceSynchronizer : ISynchronizer
{
public void SetUp()
{
 Task.Factory.StartNew(() => 
                    {
                        _logger.LogInformation($"[Watching] {location.FileSystemLocation}");

                        CustomFileSystemWatcher watcher = new CustomFileSystemWatcher(location.FileSystemLocation)
                        {
                            IncludeSubdirectories = true,
                            EnableRaisingEvents = true,
                            Filter = "",
                            CloudLocation = location.CloudLocation,
                            FileSystemErrorLocation = location.FileSystemErrorLocation,
                            NotifyFilter = NotifyFilters.DirectoryName
                                        | NotifyFilters.FileName
                                        | NotifyFilters.LastAccess
                                        | NotifyFilters.LastWrite
                        };

                        watcher.Created += OnFileCreated;
                    });
}

//_fileRead.Process: Validate and Upload to Bucket.
private void OnFileCreated(object sender, FileSystemEventArgs e)
{
worker.EnqueuedFileTask(new FileTask()
                {
                    FileRelation = new RelationLocation()
                    {
                        FileSystemLocation = e.FullPath,
                        CloudLocation = request.CloudLocation,
                        FileSystemErrorLocation = request.FileSystemErrorLocation
                    },
                    Process = _fileRead.Process
                });

}
}

Actually, I'm trying resolve this using Sqlite, where i'm send the event(file path) to database (instead of enqueue file task) and then in the Worker class try to get the paths and process. But I still losing events until nothing get any more.

UPDATE

To works all correctly finally just change my FileSystemWatcher to FileWatcherEx and preserve the pattern used here This Nuget can manage in a best way all events ocurred.

Answer 1

At least one problem I see is here:

Task.Factory.StartNew(() => 
{
    _logger.LogInformation($"[Watching] {location.FileSystemLocation}");

    CustomFileSystemWatcher watcher = new CustomFileSystemWatcher(location.FileSystemLocation)
    {
        //...
    };

    watcher.Created += OnFileCreated;
});

You are declaring a local variable which will be collected by GC on the next run effectively stopping the filewatcher, assign it to the FileServiceSynchronizer field (and make sure that FileServiceSynchronizer instance is not GCed too):

public class FileServiceSynchronizer : ISynchronizer
{
    private CustomFileSystemWatcher _watcher;

    public void SetUp()
    {
        _watcher = new CustomFileSystemWatcher(location.FileSystemLocation)
        // ...
    }
}

Also there is no reason to use Task.Factory.StartNew here.