CODEWITHSUNDEEP
google-app-enginegoogle-app-engine-pythongoogle-tasks-apigoogle-tasks
Jack tileman
Jack tileman

Reputation: 871

Exceeded soft memory limit of 512 MB with 532 MB after servicing 3 requests total. Consider setting a larger instance class in app.yaml

We are on Google App engine standard environment, F2 instance (generation 1 - python 2.7). We have a reporting module that follows this flow.

Worker Task is initiated in a queue.

    task = taskqueue.add(
            url='/backendreport',
            target='worker',
            queue_name = 'generate-reports',
            params={
                "task_data"     : task_data
            })

In the worker class, we query Google datastore and write the data to a Google Sheet. We paginate through the records to find additional report elements. When we find additional page, we call the same task again to spawn another write, so it can fetch the next set of report elements and write them to Google sheet.

in the backendreport.py we have the following code.

 class BackendReport(): 
        # Query google datastore to find the records(paginated)  
        result = self.service.spreadsheets().values().update(
                spreadsheetId=spreadsheet_Id, 
                range=range_name,
                valueInputOption=value_input_option, 
                body=resource_body).execute()

        # If pagination finds additional records
        task = taskqueue.add(
                url='/backendreport',
                target='worker',
                queue_name = 'generate-reports',
                params={
                    "task_data"     : task_data
                })

We run the same BackendReport (with pagination) as a front end job (not as a task). The pagination works without any error - meaning we fetch each page of records and display to the front end. But when we execute the tasks iteratively it fails with the soft memory limit issue. We were under the impression that every time a task is called (for each pagination) it should act independently and there shouldn't be any memory constraints. What are we doing wrong here?

Why doesn't GCP spin a different instance when the soft memory limit is reached - automatically (our instance class is F2). The error message says soft memory limit of 512 MB reached after servicing 3 requests total - does this mean that the backendreport module spun up 3 requests - does it mean there were 3 tasks calls (/backendreport)?

Upvotes: 2

Views: 3315

Answers (2)

yedpodtrzitko
yedpodtrzitko

Reputation: 9359

you're creating multiple tasks in Cloud Tasks, but there's no limitation for the dispatching queue there, and as the queue tries to dispatch multiple tasks at the same time, it reaches the memory limit. So the limitations you want to set in place is really max_concurrent_requests, however not for the instances in app.yaml, it should be set for the queue dispatching in queue.yaml, so only one task at a time is dispatched:

- name: generate-reports
  rate: 1/s
  max_concurrent_requests: 1

Upvotes: 0

Alex
Alex

Reputation: 5276

Why doesn't GCP spin a different instance when the soft memory limit is reached

One of the primary mechanisms for when app engine decides to spin up a new instance is max_concurrent_requests. You can checkout all of the automatic_scaling params you can configure here:

https://cloud.google.com/appengine/docs/standard/python/config/appref#scaling_elements

does this mean that the backendreport module spun up 3 requests - does it mean there were 3 tasks calls (/backendreport)?

I think so. To be sure, you can open up Logs viewer, find the log where this was printed and filter your logs by that instance-id to see all the requests it handled that lead to that point.

Upvotes: 1

Related Questions