CODEWITHSUNDEEP
pythoncelery
user1513388
user1513388

Reputation: 7451

Celery worker Queues

I currently using "Celeryd" to run my Celery workers as a daemon. My /etc/default/celeryd file contains the following: 

CELERYD_NODES="w1 w2 w3"

Which obviously starts three worker processes.

How do I configure routing to work with this configuration? e.g. 

celeryd -c 2 -l INFO -Q import

If I run celery from the command line I can specify the queue using the -Q flag. I need to tell my w1 worker process to only process tasks from the "import" queue.

Upvotes: 1

Views: 4037

Answers (3)

Jason V.
Jason V.

Reputation: 426

It's worth noting that you can use node names with the CELERYD_OPTS arguments, for example

CELERYD_OPTS="--time-limit=300 --concurrency=4 --concurrency:w3=8 -Q:w1 import"

Upvotes: -1

rohan
rohan

Reputation: 1656

You can make different workers consume from different/same queues by giving proper args in CELERYD_OPTS.

Refer this: http://celery.readthedocs.org/en/latest/reference/celery.bin.multi.html

The link is for celery multi documentation, but you can give the argument in same way to your case also.

# Advanced example starting 10 workers in the background:
#   * Three of the workers processes the images and video queue
#   * Two of the workers processes the data queue with loglevel DEBUG
#   * the rest processes the default' queue.
$ celery multi start 10 -l INFO -Q:1-3 images,video -Q:4,5 data -Q default -L:4,5 DEBUG

can be used as:

$ CELERYD_OPTS="--time-limit=300 --concurrency=8 -l INFO -Q:1-3 images,video -Q:4,5 data -Q default -L:4,5 DEBUG"

Do not create extra daemons unless required.

Hope this helps.

Upvotes: 7

Muatik
Muatik

Reputation: 4171

You can use the directive named CELERYD_OPTS to add optional command line arguments.

# Names of nodes to start
#   most will only start one node:
CELERYD_NODES="w1 w2 w3"

# Extra command-line arguments to the worker
CELERYD_OPTS="--time-limit=300 --concurrency=4 -Q import"

But as far as I know this option will tell all the workers will consume from only the import queue.

If you cannot find an acceptable answer, you may try to run workers separately.

Upvotes: 0

Related Questions