CODEWITHSUNDEEP
celerydjango-celerycelery-task
Cerin
Cerin

Reputation: 64689

Using Celery queues with multiple apps

How do you use a Celery queue with the same name for multiple apps?

I have an application with N client databases, which all require Celery task processing on a specific queue M.

For each client database, I have a separate celery worker that I launch like:

celery worker -A client1 -n client1@%h -P solo -Q long
celery worker -A client2 -n client2@%h -P solo -Q long
celery worker -A client3 -n client3@%h -P solo -Q long

When I ran all the workers at once, and tried to kick off a task to client1, I found it never seemed to execute. Then I killed all workers except for the first, and now the first worker receives and executes the task. It turned out that even though each worker's app used a different BROKER_URL, using the same queue caused them to steal each others tasks.

This surprised me, because if I don't specify -Q, meaning Celery pulls from the "default" queue, this doesn't happen.

How do I prevent this with my custom queue? Is the only solution to include a client ID in the queue name? Or is there a more "proper" solution?

Upvotes: 7

Views: 3398

Answers (1)

Andrey Ovcharov
Andrey Ovcharov

Reputation: 299

For multiple applications I use different Redis databases like

redis://localhost:6379/0
redis://localhost:6379/1

etc.

Upvotes: 1

Related Questions