CODEWITHSUNDEEP
djangocelerydjango-celery
S.E.T.
S.E.T.

Reputation: 137

Processor Intensive Task For Django Celery

Let's suppose, just for the sake of an example, I have a Django web application that is used for processing video and I want to use celery to offload some of those tasks. If I have a task in Celery, such as video editing, that is extremely processor intensive I would not want to run it on the same server that Django is running on. Is there a way to send a task you defined in your Django application to run remotely on another server?

I'm just now reading all the documentation about Celery and I'm not seeing this situation. The only thing that seems to fit for this situation is to run Celery somewhere else and make HTTP requests to it using the Celery HTTP Gateway.

For django-celery where do the tasks execute? Where you set your broker URL or on the machine that is running django?

Upvotes: 1

Views: 644

Answers (1)

jeorgen
jeorgen

Reputation: 656

Celery does remote task execution out of the box. Let's say you have a python module with a celery worker in it doing your video stuff, called videoprocessor.py. You place that on your remote computer and start that worker there:

celery -A videoprocessor worker

(if you want to run it from the command line)

You also place that module on the web server machine, but you do not start it there. However it is in such a place that it can be imported by the main script on your web server machine, a script called client.py for the sake of argument. Furthermore it is imported under the exact same name as on the remote machine, say something starting with "MyPackage.videoprocessor".

Now, when the main script on the web server, client.py, imports your videoprocessor.py module, it creates a signature out of the worker in that module. This signature will also be valid for videoprocessor.py on your remote computer. And since that one is running, and the videoprocessor.py on the server machine isn't, the remote worker will be used every time.

Make sure that the worker on the remote machine is using the same back end as the the client script on the web server machine. This could be Redis or RabbitMQ or something else, and you need to specify IP number at least, possibly more (database, port) depending on your setup.

Upvotes: 3

Related Questions