CODEWITHSUNDEEP
python-3.xcelerycelerybeat
Linh Nguyen
Linh Nguyen

Reputation: 3910

Celery retry only the fail request in the loop and continue with the other

I'm kinda new to celery as a whole and having the problem with retry case on a for loop:

i have the following task:

@app.task(bind=True, autoretry_for=(CustomException,), retry_kwargs={'max_retries': 10,'countdown': 30})
def call_to_apis(self):
   api_list = [api1, api2, api3, api4, api5,...]
   for api in api_list:
       try:
           response = requests.get(api)
           if response.status_code == 500:
              raise CustomException
       except CustomException:
           continue

From my understanding celery will retry on my CustomException get raised.

In the case of retry, will it only retry for the the failed api or will it just run the whole process of every api in the api_list again? If so is there anyway for it to only retry the failed api ?

Expected result: only retry the failed api

EDIT:

i have split it into 2 different tasks and 1 request function as follow:

@app.on_after_configure.connect
def setup_periodic_tasks(sender, **kwargs):
    sender.add_periodic_task(300.0, call_to_apis.s())
    print("setup_periodic_tasks")

def call_api(api):
    response = requests.get(api)
    if response.status_code == 500:
        raise CustomException
    elif response.status_code == 404:
        raise CustomWrongLinkException


@app.task(default_retry_delay=30, max_retries=10)
def send_fail_api(api):
    try:
        call_api(api)
    except NonceTooLowException:
        try:
            send_fail_api.retry()
        except MaxRetriesExceededError:
            print("reached max retry number")
            pass
    except Exception:
        pass


@app.task()
def call_to_apis():
   api_list = [api1, api2, api3, api4, api5,...]
   for api in api_list:
       try:
          call_api(api)
       except CustomException:
          send_fail_api.delay(api)
       except CustomWrongLinkException:
          print("wrong link")
       except Exception:
          pass

it worked and the other apis get to complete, with failed api it supposed to call to another task and retry for 10 time each delay 30 seconds.

But i'm getting more than expected retries about 24 times(expected to only retry 10 times) and it also printed out reached max retry number at 10th retry but it still retry until 24 retries

What am i doing wrong ?

Upvotes: 0

Views: 1144

Answers (1)

Tomáš Linhart
Tomáš Linhart

Reputation: 10220

The whole task will be retried in case of a known exception (specified in autoretry_for decorator argument), see the documentation. Celery can't know by any means the state of the task when exception is raised, that's what you have to handle. I would suggest splitting the task into individual tasks (one per API) and call them separately, presumably creating some workflow.

Upvotes: 0

Related Questions