CODEWITHSUNDEEP
celerydjango-celery
Parand
Parand

Reputation: 106380

Celery: per task concurrency limits (# of workers per task)?

Is it possible to set the concurrency (the number of simultaneous workers) on a per-task level in Celery? I'm looking for something more fine-grained that CELERYD_CONCURRENCY (that sets the concurrency for the whole daemon).

The usage scenario is: I have a single celerlyd running different types of tasks with very different performance characteristics - some are fast, some very slow. For some I'd like to do as many as I can as quickly as I can, for others I'd like to ensure only one instance is running at any time (ie. concurrency of 1).

Upvotes: 24

Views: 15400

Answers (1)

mher
mher

Reputation: 10846

You can use automatic routing to route tasks to different queues which will be processed by celery workers with different concurrency levels.

celeryd-multi start fast slow -c:slow 3 -c:fast 5

This command launches 2 celery workers listening fast and slow queues with 3 and 5 concurrency levels respectively.

CELERY_ROUTES = {"tasks.a": {"queue": "slow"}, "tasks.b": {"queue": "fast"}}

The tasks with type tasks.a will be processed by slow queue and tasks.b tasks by fast queue respectively.

Upvotes: 34

Related Questions