CODEWITHSUNDEEP
pythonrediscelery
Vedant Agarwala
Vedant Agarwala

Reputation: 18819

How can I disable redelivery of tasks in celery (with redis)?

Is there for me to configure celery to just drop the tasks in case of a non-graceful shutdown of a worker? Its more critical for me that tasks are not repeated rather than they are always delivered.

As mentioned in the docs:

If a task isn’t acknowledged within the Visibility Timeout the task will be redelivered to another worker and executed.

This causes problems with ETA/countdown/retry tasks where the time to execute exceeds the visibility timeout; in fact if that happens it will be executed again, and again in a loop.

So you have to increase the visibility timeout to match the time of the longest ETA you’re planning to use.

My use case is that I am using a visibility_timeout of 1 day, but still in some cases that is not enough- I want to schedule tasks even further in the future. "Power failure" or any other event causing a non-graceful shutdown is very rare and I'm fine with tasks being dropped in, say, 0.01% of the cases. Moreover, a task executed 1 day later than it was supposed to, is as bad as the task not being run at all.

One obvious, hacky, way is to set visibility_timeout to 100 years. Is there a better way?

Upvotes: 1

Views: 1671

Answers (1)

ItayB
ItayB

Reputation: 11337

There's a acks_late configuration, but the default value is false (so make sure you didn't enable it):

The acks_late setting would be used when you need the task to be executed again if the worker (for some reason) crashes mid-execution. It’s important to note that the worker isn’t known to crash, and if it does it’s usually an unrecoverable error that requires human intervention (bug in the worker, or task code).

(quote from here)

The definition of task_acks_late (seems like the name has changed in the last version of some mismatch) can be found here.

Upvotes: 1

Related Questions