CODEWITHSUNDEEP
celerydjango-celerycelery-task
Pooja
Pooja

Reputation: 796

Celery configuration for retry backoff

I want to have all asycnhronous tasks in my app retry on any exception and also want the retries to follow exponential backoff. 

@celery_app.task(autoretry_for=(Exception,))
def some_task():
    ...

In my configuration I have

CELERY_TASK_ANNOTATIONS = {'*': {'max_retries': 5, 'retry_backoff': 5}}

The max_retries setting works and all tasks are now retried 5 times before failing. But all of them are retried after 180 seconds. I want some way for all the tasks to follow retry_backoff without having to specify it for each of them so that I can change it anytime at one place.

Upvotes: 2

Views: 4681

Answers (2)

Hungry Beast
Hungry Beast

Reputation: 3725

It looks like according to the Celery documentation the property you want to set is retry_backoff_max.

Task.retry_backoff_max

A number. If retry_backoff is enabled, this option will set a maximum delay in seconds between task autoretries. By default, this option is set to 600, which is 10 minutes.

retry_backoff can be a number or a boolean and based on which it is the backoff will behave differently. For an exponential backoff it appears you want to set this true.

Task.retry_backoff

A boolean, or a number. If this option is set to True, autoretries will be delayed following the rules of exponential backoff. The first retry will have a delay of 1 second, the second retry will have a delay of 2 seconds, the third will delay 4 seconds, the fourth will delay 8 seconds, and so on. (However, this delay value is modified by retry_jitter, if it is enabled.) If this option is set to a number, it is used as a delay factor. For example, if this option is set to 3, the first retry will delay 3 seconds, the second will delay 6 seconds, the third will delay 12 seconds, the fourth will delay 24 seconds, and so on. By default, this option is set to False, and autoretries will not be delayed.

Upvotes: 3

DejanLekic
DejanLekic

Reputation: 19787

What you can do to avoid changing this in multiple places is to have a global variable, say global_retry_backoff=5 that you will use in your task annotations: @celery_app.task(autoretry_for=(Exception,), retry_backoff=global_retry_backoff) .

Upvotes: 1

Related Questions