CODEWITHSUNDEEP
pythondjangocelerydjango-celery
dr_rompecabezas
dr_rompecabezas

Reputation: 61

How to retry Django Celery task on Exception when Internal Server Error is raised?

I am trying to use Celery to send anywhere from 1 to about 25 consecutive requests to a third-party API. The measure of success is whether I get a URL back in the response payload: response_json["data"]["url"]. Celery or not, sometimes I get the data I expect, and sometimes I don't.

I decided to experiment with Celery to retry the API call while taking advantage of the built-in exponential backoff, which sounds perfectly suited to my needs, but I am struggling to implement.

When the data I am expecting is not available in the response payload, I get a TypeError: 'type' object is not iterable (i.e., there is no such item in the response). But I am also getting an Internal Server Error, which I wonder whether I need to handle or whether I could use to trigger the retry.

Here is a example of one of several similar approaches I have tried:

@shared_task(autoretry_for=(Exception), retry_backoff=True, retry_backoff_max=120)
def third_party_api_request(payload, api_url, headers):
    response = requests.request("POST", api_url, headers=headers, data=payload)
    response_json = response.json()
    return response_json["data"]["url"]

# output:
Internal Server Error:
-- snip --
    autoretry_for = tuple(
TypeError: 'type' object is not iterable

Another approach I have tried:

@shared_task(bind=True)
def third_party_api_request(self, payload, api_url, headers):
    try:
        response = requests.request("POST", api_url, headers=headers, data=payload)
        response_json = response.json()
        return response_json["data"]["url"]
    except TypeError as exc:
        logger.error("Error sending request to API: %s", exc)
        raise self.retry(exc=exc)

# output:
ERROR 2022-04-22 17:31:40,131 tasks 72780 123145528369152 Error sending request to API: 'NoneType' object is not subscriptable
Internal Server Error:
-- snip --
TypeError: 'NoneType' object is not subscriptable
-- snip --
    raise ret
celery.exceptions.Retry: Retry in 180s: TypeError("'NoneType' object is not subscriptable")
ERROR 2022-04-22 17:31:40,409 log 72780 123145528369152 Internal Server Error:
-- snip --
TypeError: 'NoneType' object is not subscriptable
-- snip --
   raise ret
celery.exceptions.Retry: Retry in 180s: TypeError("'NoneType' object is not subscriptable")

And yet another approach, with similar results:

@shared_task(autoretry_for=(TypeError), retry_backoff=True, retry_backoff_max=120)
def send_http_request_to_proctoru_task(payload, api_url, headers):
    response = requests.request("POST", api_url, headers=headers, data=payload)
    response_json = response.json()
    try:
        return response_json["data"]["url"]
    except TypeError:
        logger.error("API response: %s", response_json)
        raise

Upvotes: 1

Views: 1359

Answers (1)

dr_rompecabezas
dr_rompecabezas

Reputation: 61

The issue was I had misplaced the return statement. I still have much fine-tuning to do, but the following code solves the issue in the question that I posed earlier today. I have been reading posts, documentation, and articles for days, and I wish I could everyone credit, but this is the blog post that ultimately helped me realize my error: https://testdriven.io/blog/retrying-failed-celery-tasks/.

@shared_task(name="send_http_request_to_proctoru_task", bind=True, max_retries=6)
def send_http_request_to_proctoru_task(self, api_url, headers, payload):
    try:
        response = requests.request("POST", api_url, headers=headers, data=payload)
        response_json = response.json()
        if response_json["response_code"] == 2:
            raise Exception()

        return response_json["data"]["url"]
    except Exception as exc:
        logger.warning("Exception raised. Executing retry %s" % self.request.retries)
        raise self.retry(exc=exc, countdown=2 ** self.request.retries)

Upvotes: 2

Related Questions