CODEWITHSUNDEEP
rabbitmqcelery
Piotr Zakrzewski
Piotr Zakrzewski

Reputation: 3891

Celery task with a long ETA and RabbitMQ

RabbitMQ may enforce ack timeouts for consumers: https://www.rabbitmq.com/consumers.html#acknowledgement-modes By default if a task has not been acked within 15 min entire node will go down with a PreconditionFailed error. I need to schedule a celery task (using RabbitMQ as a broker) with an ETA quite far in the future (1-3 h) and as of now (with celery 4 and rabbitmq 3.8) when I try that... I get PreconditionFailed after the consumer ack timeout configured for my RMQ. I expected that the task would be acknolwedged before its ETA ...

Is there a way to configure an ETA celery task to be acknowledged within the consumer ack timeout?

right now I am increasing the consumer_timeout to above my ETA time delta, but there must be a better solution ...

Upvotes: 6

Views: 10078

Answers (5)

Anandhu Gopi
Anandhu Gopi

Reputation: 61

I also wanted to do something similar and have tried something with the "rabbitmq-delayed-exchange-plugin" and "dead-letter-queue". I wrote an article about both and mentioned the links below. I hope it will be helpful to someone. In a nutshell, we can use both approaches for scheduling celery tasks( handling long ETA).

using dlx:

Dead Letter Exchanges (DLX): enter image description here

using RabbitMQ Delayed Message Plugin:

RabbitMQ Delayed Message Plugin: enter image description here

p.s: I know StackOverflow answers should be self-explanatory, but posting the links as answers is long.Sorry

Upvotes: 0

Sarang
Sarang

Reputation: 2713

There's a way to change this consumer_timeout for a running instance by running the following command on the RabbitMQ server:

rabbitmqctl eval 'application:set_env(rabbit, consumer_timeout, 36000000).'

This will set the new timeout to 10 hrs (36000000ms). For this to take effect, you need to restart your workers though. Existing worker connections will continue to use the old timeout.

You can check the current configured timeout value as well:

rabbitmqctl eval 'application:get_env(rabbit, consumer_timeout).'

If you are running RabbitMQ via Docker image, here's how to set the value: Simply add -e RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS="-rabbit consumer_timeout 36000000" to your docker run OR set the environment RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS to "-rabbit consumer_timeout 36000000".

Hope this helps!

Upvotes: 8

PicxyB
PicxyB

Reputation: 837

I encountered the same problem and I resolved it.

With RabbitMQ version 3.8.14 (3.8.14-management), I am able to send long ETA tasks.

I personaly use Celery to send tasks with a long ETA. In my case, I setup up celery to add a timeout (~consumer_timeout), I can configure it with time_limit or soft_time_limit

Upvotes: 0

Ali Emamalinezhad
Ali Emamalinezhad

Reputation: 1

I faced this problem, actually i think you would better to use PeriodicTask, if you would like only do it once set the one_off=True.

https://docs.celeryq.dev/en/stable/userguide/periodic-tasks.html?highlight=periodic

Upvotes: 0

Chad Knutson
Chad Knutson

Reputation: 391

I think adjusting the consumer_timeout is your only option in Celery 5. Note that this is only applicable for RabbitMQ 3.8.15 and newer.

Another possible solution is to have the workers ack the message immediately upon receipt. Do this only if you don't need to guarantee task completion. For example, if the worker crashes before doing the task, Celery will not know that it wasn't completed.

In RabbitMQ, the best options for delayed tasks are the delayed-message-exchange or dead lettering. Celery cannot use either option. In Celery, messages are published to the message broker where they are sent to consumers as soon as possible. The delay is enforced in the worker, not at the broker.

Upvotes: 5

Related Questions