CODEWITHSUNDEEP
celerycelery-task
Alec Gerona
Alec Gerona

Reputation: 2897

In Celery tasks, do retry_backoff and retry_backoff_max affect the manual self.retry() calls?

So according to the docs, default_retry_delayspecifies the time in seconds before a retry of the task should be executed. I just want to be sure that this only affects the manually called self.retry()calls and not the autoretries triggered by Celery when the task encounters predefined exceptions.

Likewise, I want to know if retry_backoff and retry_backoff_maxonly affect the autoretries and not the manual self.retry().

Finally, what happens when all of these are set?

Upvotes: 3

Views: 1168

Answers (1)

tbrknt
tbrknt

Reputation: 25

That answer may change depending on the version of Celery that is used. However, as far as I have checked with the source code of versions v4.3.1, v4.4.7 and v5.0.0, retry_backoff and retry_backoff_max only affect the auto-retry. If autoretry_for is not set as True (or a True value such as positive integers), then the backoff parameters is not used.

As it can be seen from the implementation, when a True value is set to autoretry_for, Celery uses retry_backoff, retry_backoff_max, and retry_jitter arguments in order to compute the countdown of the next retry, before calling .retry method. Thus, these retry_xx arguments are not directly used by the .retry method; however, they affect the value of countdown. Thus, they are only used in the auto_retry.

When the countdown is explicity defined or computed via autoretry, then the value of default_retry_delay is ignored, as it can be seen in the implementation of v5.0.0. The same is valid when eta is set. Thus, to be able to use default_retry_delay, countdown, and eta should not be set. Additionally, autoretry_for should not be set or it should be set with a False value.

Another approach to use default_retry_delay and autoretry at the same time would be to define autoretry and retry variables; but for manual retry calls, set the eta and countdown to None explicitly. One example:

@app.task(bind=True, default_retry_delay=360, autoretry_for=[ExternalApiError], max_retries=5, retry_backoff=1800, retry_backoff_max=8 * 3600, retry_jitter=False)
def my_task(self, data):
   try:
      ... # some code here that may raise ExternalApiError
   except SomeOtherError:
      try:
         self.retry(countdown=None, eta=None)
      except self.MaxRetriesExceededError:
         ... # when maximum number of retries has been exceeded.

If you want to use backoff parameters with manual retry calls, it can be implemented by using the method get_exponential_backoff_interval of Celery. One example could be this:


from celery.utils.time import get_exponential_backoff_interval

@app.task(bind=True, max_retries=5)
def my_task(self, data):
   try:
      ... # some code here that may raise ExternalApiError
   except SomeOtherError:
      try:
         retry_backoff = True
         retry_backoff_max = 5
         retry_jitter = True
         countdown = get_exponential_backoff_interval(
             factor=retry_backoff,
             retries=self.request.retries,
             maximum=retry_backoff_max,
             full_jitter=retry_jitter)
         )
         self.retry(countdown=countdown)
      except self.MaxRetriesExceededError:
         ... # when maximum number of retries has been exceeded.

Upvotes: 2

Related Questions