CODEWITHSUNDEEP
rabbitmqcelerydjango-celery
Himberjack
Himberjack

Reputation: 5802

Celery with rabbitmq creates results multiple queues

I have installed Celery with RabbitMQ. Problem is that for every result that is returned, Celery will create in the Rabbit, queue with the task's ID in the exchange celeryresults.

I still want to have results, but on ONE queue.

my celeryconfig:

from datetime import timedelta
OKER_URL = 'amqp://'
CELERY_RESULT_BACKEND = 'amqp'
#CELERY_IGNORE_RESULT = True
CELERY_TASK_SERIALIZER = 'json'
CELERY_RESULT_SERIALIZER = 'json'
CELERY_ACCEPT_CONTENT=['json', 'application/json']
CELERY_TIMEZONE = 'Europe/Oslo'
CELERY_ENABLE_UTC = True

from celery.schedules import crontab

CELERYBEAT_SCHEDULE = {
    'every-minute': {
        'task': 'tasks.remote',
        'schedule': timedelta(seconds=30),
        'args': (),
    },
}

Is that possible? How?

Thanks!

Upvotes: 10

Views: 4176

Answers (4)

Becca Petrin
Becca Petrin

Reputation: 1644

This solution won't keep all the results to ONE queue, but it will at least clean up the extra queues right when you're done with them.

If you use Redis as your backend, when you're done with a result that has created an errant queue, run result.forget(). This will cause both the result and the queue for the result to disappear. This can help you manage the number of queues you have, and prevent OOM issues.

Upvotes: 0

rohan
rohan

Reputation: 1656

Nothing unusual.

That is how celery works when we use amqp as result backend. It will create a new temporary queue for every result corresponding to each tasks that worker consumes.

If you are not interested in the result, you can try CELERY_IGNORE_RESULT = True setting

If you do want to store the result, then i would recommend using a different result backend like Redis.

Upvotes: 6

Zoran Pavlovic
Zoran Pavlovic

Reputation: 1232

You say you want Celery to keep the result on one queue. Now, to answer your question, let me ask you one:

How do you expect each producer to check for it's relevant result without reading every single message off the queue to find the one it needs/wants?

In essence, what you want is a database of key-value pairs so that the lookup is O(1). The only way to do that with a queue broker is to create one queue for each "pair".

I understand that having many GUID queues is not neat or pretty, but it's conceptually the only way to do it on a messaging broker.

Upvotes: 2

mher
mher

Reputation: 10846

amqp backend creates a new queue for each task. Alternatively, there is a new rpc backend which keeps results in a single queue.

http://docs.celeryproject.org/en/master/whatsnew-3.1.html#new-rpc-result-backend

Upvotes: 9

Related Questions