CODEWITHSUNDEEP
pythonceleryinitceleryd
Jason
Jason

Reputation: 11363

Multiple queue for Celery daemon

I have a network of Celery servers and workers that are to be used for a high volume I/O task coming up. There are two queues, default and backlog, and each server has five workers. All servers are daemonized with a configuration much like the init script config documentation.

What I'd like to do for one server is have three workers for default and two for backlog. Is it possible to do this with a daemon configuration?

Upvotes: 2

Views: 1775

Answers (1)

Mauro Rocco
Mauro Rocco

Reputation: 5128

Have a look here in this part whare it shows you an example configuration it also says:

# Names of nodes to start
#   most people will only start one node:
CELERYD_NODES="worker1"
#   but you can also start multiple and configure settings
#   for each in CELERYD_OPTS (see `celery multi --help` for examples):

So as you can see is possible to have celeryd starting multiple nodes and you can configure each of them via CELERYD_OPTS, therefore you can set different queue for each of them.

Here you found another more complete example of celery multi, I post here a little extract.

# Advanced example starting 10 workers in the background:
#   * Three of the workers processes the images and video queue
#   * Two of the workers processes the data queue with loglevel DEBUG
#   * the rest processes the default' queue.
$ celery multi start 10 -l INFO -Q:1-3 images,video -Q:4,5 data
    -Q default -L:4,5 DEBUG

Upvotes: 2

Related Questions