Long running, polling, queueing process for Python. What's the best stuff to use?

Question

Feel free to close and/or redirect if this has been asked, but here's my situation:

I've got an application that will require doing a bunch of small units of work (polling a web service until something is done, then parsing about 1MB worth of XML and putting it in a database). I want to have a simple async queueing mechanism that'll poll for work to do in a queue, execute the units of work that need to be done, and have the flexibility to allow for spawning multiple worker processes so these units of work can be done in parallel. (Bonus if there's some kind of event framework that would also me to listen for when work is complete.)

I'm sure there is stuff to do this. Am I describing Twisted? I poked through the documentation, I'm just not sure exactly how my problems maps onto their framework, but I haven't spent much time with it. Should I just look at the multiprocess libraries in Python? Something else?

Answer 1

This problem sounds like a pretty good candidate for Python's built-in (2.6+ anyway) multiprocessing module: http://docs.python.org/library/multiprocessing.html

A simple solution would be to create a process Pool and use your main program to poll for the XML chunks. Once it has them it can then pass them off to the pool for parsing/persisting.

Answer 2

There's celery.

You could break it down into 2 different Tasks: one to poll the web service and queue the second task, which would be in charge of parsing the XML and persisting it.