CODEWITHSUNDEEP
pythoninotify
Jon Martin
Jon Martin

Reputation: 3392

Files appearing in directory faster than I can process them (pyinotify)

I'm using pyinotify (the process_IN_MOVED event) to watch for files appearing in a directory. When a new file appears, this event will be triggered and the file will be processed in some way. The problem is that sometimes the files appear in the directory faster than I can process them, meaning that a bunch of files won't get processed. I can just have a function to sleep for ten seconds and then wake up to find new files or something, but I really want to stick to the event-based solution if possible. Is there a way to do this?

Upvotes: 4

Views: 369

Answers (3)

TimStaley
TimStaley

Reputation: 535

As Humungus notes, a thread pool is a good option here.

I've just posted some code I wrote for exactly this problem, at:

https://github.com/timstaley/autocrunch

It's a bit cluttered with application specific details but you should be able to drop in replacement functions for your own needs. I might write a blog post with a cut down version at some point, but don't have time right now. HTH!

Upvotes: 1

Humungus
Humungus

Reputation: 585

Generally I would implement a thread pool here to handle the processing while the event watcher would be just watching for the events and passing them to the pool. Rough example:

(event happens) -> 
Watcher registers the event -> 
puts it into the thread pool queue -> 
thread pool processes the event

That way the watcher will spend minimum time outside of the waiting part, thus greatly reducing the chance of missing an update.

Upvotes: 3

Julian
Julian

Reputation: 3429

Twisted has inotify support. You can give it a callback to do your processing. You definitely don't want to be sleeping. It'd depend on what kind of processing you're doing as to whether you want to do it in-process or via another process, but you shouldn't be losing events.

Upvotes: 2

Related Questions