CODEWITHSUNDEEP
pythongoogle-app-enginegoogle-compute-enginetask-queue
xmedeko
xmedeko

Reputation: 7820

Google Task Queue REST pull returning 500 occasionally

I have Python a process leasing tasks from the Google TaskQueue REST API every second in the unlimited loop:

credentials = GoogleCredentials.get_application_default()
task_api = googleapiclient.discovery.build('taskqueue', 'v1beta2', credentials=credentials)
while True:
    tasks = task_api.tasks().lease(...).execute()
    time.sleep(1)

The process sometimes run well for hours. But occasionally crashes often by one of HTTP error:

The process is running on the Google Computing Engine server. It uses a service account key, specified by the GOOGLE_APPLICATION_CREDENTIALS env variable. Is this a Google Task Queue bug or do I miss something? E.g. do I need to reread the credentials before every lease request?

Upvotes: 2

Views: 117

Answers (1)

xmedeko
xmedeko

Reputation: 7820

Since the @DalmTo has just answered in comments, I sum up his answers and add the Python solution.

The Google 5xx backed error is flood protection and Google recommends to implement exponential backoff. Despite the link points to Google Drive API, the Google errors are system wide for all the APIs (GAE including). It rarely takes more then 6 retries for it to kick in and respond.

After digging the googleapiclient sources, I've found that the exponential backoff is already implemented in this library, so the solution is dead simple:

tasks = task_api.tasks().lease(...).execute(num_retries=6)

The sources of the http.py::_should_retry_response() shows, that beside HTTP 5xx errors the request is repeated also when the JSON response contains userRateLimitExceeded or rateLimitExceeded error.

Upvotes: 2

Related Questions