CODEWITHSUNDEEP
pythonqueuecelerypriority-queue
FMc
FMc

Reputation: 42421

Does a Celery worker check queues in a specific order?

If a Celery worker consumes from multiple queues:

celery -A proj worker -Q foo,bar

Does it check the queues in a specific order -- perhaps in the order they are listed on the command line (first check foo and then check bar)? I hope so.

I have been unable to find documentation that clarifies this point (for example, here) and would appreciate a more definitive answer.

Upvotes: 1

Views: 705

Answers (1)

FMc
FMc

Reputation: 42421

Based on some experiments, it looks like Celery does not check queues in any particular order (at least when using Redis as a broker):

# demo.py
#
# Usage:
#
#    Add some work to the queues.
#
#        python demo.py add add mul mul add
#
#    Start up Celery:
#
#        celery worker -A demo --queues add,mul

from __future__ import absolute_import

import sys
from celery import Celery

app = Celery('demo', broker  = 'redis://localhost:6379/0')
app.conf.update(CELERY_ACCEPT_CONTENT = ['pickle'])

@app.task
def add(x, y):
    print 'add({}, {}) = {}'.format(x, y, x + y)

@app.task
def mul(x, y):
    print 'mul({}, {}) = {}'.format(x, y, x * y)

def main(qs):
    for q in qs:
        func = add if q == 'add' else mul
        for i in xrange(5, 8):
            func.apply_async((i, i), queue = q)

if __name__ == '__main__':
    main(sys.argv[1:])

Example output:

# [2014-08-31 13:15:33,005: WARNING/Worker-2] mul(5, 5) = 25
# [2014-08-31 13:15:33,005: WARNING/Worker-8] add(5, 5) = 10
# [2014-08-31 13:15:33,007: WARNING/Worker-4] add(6, 6) = 12
# [2014-08-31 13:15:33,007: WARNING/Worker-3] mul(6, 6) = 36
# [2014-08-31 13:15:33,009: WARNING/Worker-5] add(7, 7) = 14
# [2014-08-31 13:15:33,009: WARNING/Worker-7] mul(7, 7) = 49
# [2014-08-31 13:15:33,011: WARNING/Worker-8] mul(5, 5) = 25
# [2014-08-31 13:15:33,011: WARNING/Worker-1] add(5, 5) = 10
# [2014-08-31 13:15:33,013: WARNING/Worker-4] mul(6, 6) = 36
# [2014-08-31 13:15:33,013: WARNING/Worker-6] add(6, 6) = 12
# [2014-08-31 13:15:33,015: WARNING/Worker-5] mul(7, 7) = 49
# [2014-08-31 13:15:33,015: WARNING/Worker-2] add(7, 7) = 14
# [2014-08-31 13:15:33,016: WARNING/Worker-1] add(5, 5) = 10
# [2014-08-31 13:15:33,017: WARNING/Worker-3] add(6, 6) = 12
# [2014-08-31 13:15:33,018: WARNING/Worker-6] add(7, 7) = 14

Upvotes: 3

Related Questions