Reputation: 745
Is it possible to have multiple celery instances possible on different machines consuming from a single queue for tasks, working with django preferably using django-orm as the backend? How can i implement this if possible, I can't seem to find any documentation for this.
Upvotes: 0
Views: 2676
Reputation: 5205
Yes it's possible, they just have to use the same broker. For instance, if you are using AMQP, the configs on your servers must share the same
BROKER_URL = 'amqp://user:password@localhost:5672//'
See the routing page for more details. For instance let's say you want to have a common queue for two servers, then one specific to each of them, you could do
CELERY_ROUTES = {'your_app.your_specific_tasks1': {'queue': 'server1'}}
user@server1:/$ celery -A your_celery_app worker -Q server1, default
CELERY_ROUTES = {'your_app.your_specific_tasks2': {'queue': 'server2'}}
user@server2:/$ celery -A your_celery_app worker -Q server2, default
Of course it's optional, by default all the tasks will be routed to the queue named celery
.
Upvotes: 1