CODEWITHSUNDEEP
pythongoogle-app-enginememory-managementpython-2.7task-queue
Andrew F
Andrew F

Reputation: 595

Restricting the number of task requests per App Engine instance

I have a Google App Engine app that periodically processes bursts of memory-intensive long-running tasks. I'm using the taskqueue API on the Python2.7 run-time in thread-safe mode, so each of my instances is handling multiple tasks concurrently. As a result, I frequently get these errors:

Exceeded soft private memory limit with 137.496 MB after servicing 8 requests total

After handling this request, the process that handled this request was found to be using too much memory and was terminated. This is likely to cause a new process to be used for the next request to your application. If you see this message frequently, you may have a memory leak in your application.

As far as I can tell, each instance is taking on 8 tasks each and eventually hitting the soft memory limit. The tasks start off using very low amounts of memory but eventually grow to about 15-20MB.

Is there any way to restrict tell App Engine to assign no more than 5 requests to an instance? Or tell App Engine that the task is expected to use 20MB of memory over 10 minutes and to adjust accordingly?

I'd prefer not to use the backend APIs since I want the number of instances handling tasks to automatically scale, but if that's the only way, I'd like to know how to structure that.

Upvotes: 1

Views: 241

Answers (1)

Nick Johnson
Nick Johnson

Reputation: 101149

There's not currently any way to advise the App Engine infrastructure about this. You could have your tasks return a non-200 status code if they shouldn't run now, in which case they'll be automatically retried (possibly on another instance), but that could lead to a lot of churn. Backends are probably your best option.

If you set up dynamic backends, they'll only be spun up as required for task queue traffic. You can send tasks to a backend by specifying the URL of the backend as the 'target' argument.

You can gain even more control over task execution by using pull queues. Then, you can spin up backends as you choose (or use push queue tasks, for that matter), and have the instances pull tasks off the pull queue in whatever manner suits.

Upvotes: 7

Related Questions