CODEWITHSUNDEEP
pythontornado
NetRocks
NetRocks

Reputation: 477

Tornado Subprocess confusion

I am trying to build a Tornado web server which takes requests from multiple clients. The request consists of: a. For a given directory name passed through an URL, zip the files, etc and FTP it out. b. Providing a status of sorts if the task is completed.

So, rather than making it a synchronous and linear process, I wanted to break it down into multiple subtasks. The client will submit the URL request and then simply receive a response of sorts 'job submitted'. A bit later, the client can come along asking status on this job. During this time the job obviously has to finish its task.

I am confused between what modules to use - Tornado Subprocess, Popen contructor, Subprocess.Call, etc. I've read Python docs but can't find anything where the task is running longer and Tornado is not supposed to wait for it to finish. So, I need a mechanism to start a job, let it run its course but relinquish the client and then when asked by client provide a status on it. Any help is appreciated. Thanks.

Upvotes: 1

Views: 504

Answers (1)

A. Jesse Jiryu Davis
A. Jesse Jiryu Davis

Reputation: 24007

Python programmers widely use Celery for a set of processes to manage a queue of tasks. Set up Celery with RabbitMQ and write a Celery worker (perhaps with Celery Canvas that does the work you need: zips a directory, ftps it to somewhere, etc.

The Tornado-Celery integration package provides something that appears close to what you need to integrate your Tornado application with Celery.

This is all a lot of moving parts to install and configure at first, of course, but it will prepare you for a maintainable application architecture.

Upvotes: 1

Related Questions