CODEWITHSUNDEEP
c#.netiissystem.reactive
GregC
GregC

Reputation: 8007

How to monitor FTP incoming folder for newly uploaded files?

In other words, how can I kick off some custom processing of newly uploaded files to FTP server on my machine with IIS-backed FTP server and .NET?

Clarification This is to run on the machine that hosts FTP server. Permissions are not an issue in my case.

It seems I'd need to ask the uploading entities to add a listing file that is sent after the actual data file is completely uploaded. Imagine that there is an actual xxx.data file uploaded, followed by xxx.listing file that signifies completion of xxx.data file upload.

Originally this question was talking about Reactive Extensions. I was hoping that someone has a bridge from IIS 7.5's FTP server to Reactive Extensions, so I can focus on the higher-level implementation details.

Upvotes: 3

Views: 6447

Answers (4)

mcoolive
mcoolive

Reputation: 4205

To know that a new file is complete, a frequent pattern is that the client uploads the file in a TEMP directory, then it does an FTP move command. The move (or a rename) is atomic IF the source and target directories are on the same partition.

Upvotes: 1

GregC
GregC

Reputation: 8007

I've slimmed down the MSDN sample to focus in on file creation, as it's the only event of interest when files are uploaded to FTP server.

One would have to install the Reactive Extensions for this code sample to work in Visual Studio 2012.

 class Program
 {
     static void Main()
     {
         // Create a FileSystemWatcher to watch the FTP incoming directory for creation of listing file

         using (var watcher = new FileSystemWatcher(@"C:\FTP-Data\Incoming", "*.lst"))
         {
             // Use the FromEvent operator to setup a subscription to the Created event.           
             //                                                                                    
             // The first lambda expression performs the conversion of Action<FileSystemEventArgs> 
             // to FileSystemEventHandler. The FileSystemEventHandler just calls the handler       
             // passing the FileSystemEventArgs.                                                   
             //                                                                                    
             // The other lambda expressions add and remove the FileSystemEventHandler to and from 
             // the event.                                                                         

             var fileCreationObservable = Observable.FromEvent<FileSystemEventHandler, FileSystemEventArgs>(
                                         UseOnlyTheSecondArgument,
                                         fsHandler => watcher.Created += fsHandler,
                                         fsHandler => watcher.Created -= fsHandler);

             fileCreationObservable.Subscribe(ActionWhenFileIsUploaded);

             watcher.EnableRaisingEvents = true;

             Console.WriteLine("Press ENTER to quit the program...\n");
             Console.ReadLine();
         }
     }

     private static void ActionWhenFileIsUploaded(FileSystemEventArgs args)
     {
         Console.WriteLine("{0} was created.", args.FullPath);

         // TODO
         // 1. Deduce original file name from the listing file info
         // 2. Consume the data file
         // 3. Remove listing file
     }

     private static FileSystemEventHandler UseOnlyTheSecondArgument(Action<FileSystemEventArgs> handler)
     {
         return (object sender, FileSystemEventArgs e) => handler(e);
     }
 }

I am still working out the details of transforming observable of listing files to observable of actual data files. Stay tuned.

Upvotes: 2

Amitd
Amitd

Reputation: 4849

You will need to extend the capabilities of FileWatcher i.e. to check if your file was completely uploaded (or not) and then use the Rx samples listed in the comments.

Or..

Not directly watch the files in FTP folder.. instead run a windows service or some app to copy the fully uploaded file to a processing folder, then you are sure that only there are no partial files. (will need to figure out how. Wonder if a file is locked for the period it is uploading the server ) . This separates watching and processing logic in separate apps.(but makes it trickier.) . Also after processing you can move the file out the processing folder so that you don't process the same file again(or you can use some other logic.)

Upvotes: 0

Enigmativity
Enigmativity

Reputation: 117124

Rx isn't a good fit, per se.

Rx is a good fit when you want to query events (or collections in time) and you want to compose complex filtering, joining and projection.

Rx does absolutely nothing with regard to watching FTP uploads or file system changes. Or any event source really. It does not provide any mechanism to generate those types of events.

Instead it does allow a way to bring together all sorts of different event sources - events, async operations, WMI events, generated observables, etc - into a common framework.

You need to look at something like FileSystemWatcher and then turn this into an observable.

Then you can do some awesome things with Rx. Truly awesome.

Upvotes: 2

Related Questions