CODEWITHSUNDEEP
pythonrediscelery
free2use
free2use

Reputation: 387

Celery prioritized queues

Ok, I've searched a little about this problem and didn't find much.
For example i have one worker and two different sources of tasks. One of them is scheduled with beat, starting one little tasks each hour, another one have enormous number of tasks, which will be going non stop for something about month. Problem is that this little tasks will never ran, or it will be with huge delay.
So, of course i tried to separate them in different queues but this makes no sense since both of them are processed by single worker (two different workers are ok, but its not the case of my question). I want to know if there is built in mechanism for prioritizing queues for celery worker. Or, at least i would like if someone will point me on some information about how do worker selects which task from which queue must be processed at that moment. With one queue it seems to be straightforward, linear. But what about different queues processed by same worker?

Thank You

Upvotes: 1

Views: 831

Answers (1)

Rustem
Rustem

Reputation: 2932

there are no any way to prioritize tasks in queue. Queue is queue, celery can just fetch task one by one. Possible solution is

http://docs.celeryproject.org/en/latest/userguide/routing.html#automatic-routing

You can start two separate workers ( with different --queue option). I think this will be enough.

Upvotes: 1

Related Questions