CODEWITHSUNDEEP
pythondjangocelerycelery-task
Shaumux
Shaumux

Reputation: 745

Multiple Celery instances consuming a single queue

Is it possible to have multiple celery instances possible on different machines consuming from a single queue for tasks, working with django preferably using django-orm as the backend? How can i implement this if possible, I can't seem to find any documentation for this.

Upvotes: 0

Views: 2676

Answers (1)

Seb D.
Seb D.

Reputation: 5205

Yes it's possible, they just have to use the same broker. For instance, if you are using AMQP, the configs on your servers must share the same

BROKER_URL = 'amqp://user:password@localhost:5672//'

See the routing page for more details. For instance let's say you want to have a common queue for two servers, then one specific to each of them, you could do

On server 1:

CELERY_ROUTES = {'your_app.your_specific_tasks1': {'queue': 'server1'}}
user@server1:/$ celery -A your_celery_app worker -Q server1, default

On server 2:

CELERY_ROUTES = {'your_app.your_specific_tasks2': {'queue': 'server2'}}
user@server2:/$ celery -A your_celery_app worker -Q server2, default

Of course it's optional, by default all the tasks will be routed to the queue named celery.

Upvotes: 1

Related Questions