CODEWITHSUNDEEP
c#filedirectoryfilesystems
Hugo Mota
Hugo Mota

Reputation: 11577

What's a fast way to scan for directory changes without monitoring?

I'm writing an application in which I have to detect file changes inside a directory. New files, missing files, and so on. Each scan relative to the previous.

I'm currently doing a recursive scan to retrieve all the paths and comparing to the previous list in my database. The problem with this is that some clients require scanning for millions of files. This makes the system consume a lot of resources (cpu and memory). I'm even getting SystemOutOfMemory exceptions.

So I'm wondering if there's a better way to find those changes, maybe without a full scan.

Important note: I can't "monitor" for events since I have to detect changes that happened between scans, no matter if the system was running. I can't afford to lose a single change. So, unless it can catch changes made while it wasn't running I can't use FileSystemWatcher for this.

Upvotes: 0

Views: 440

Answers (1)

user6492769
user6492769

Reputation:

Without knowing how the Directory you are scanning is structured im assuming you have something simmilar to following example.

000/
001/
002/

With each directory having subdirectories simmilar to the top level directory.

This would allow you to build up an index like git does internally. Storing identifiers for each tree item (subdirectories, Files) for easy comparrison.

You should then be able to counter the SystemOutOfMemoryException through splitting up the task into multiple sub tasks.

Regarding the runtime i see no possibility to lower it as one component of your system will always have to watch or compare each Item.

If when the files are written the modify date of the directories ist reliably updated you could use this as part of your comparrison.

Upvotes: 1

Related Questions