CODEWITHSUNDEEP
google-app-enginegoogle-cloud-datastore
user1532708
user1532708

Reputation: 41

Google App Engine App get terminated due to memory limit (at 40MB usage)?

My GAE backend always get terminated at 40MB memory usage and log said it used too much memory. I don't know why since I used B1/B2 which should have 128/256MB.

Here is the details

I create my GAE app with a frontend module and a backend (in python). The frontend gets the user requests and send to backend. Backend is a image processing app involving google cloud storage service. It needs a long time to finish and consumes some memory as expected. The backend module run for a long time (5-10mins maybe) so I start the function as a background thread like shown in document:

t = background_thread.BackgroundThread(target=f, args=["foo", "bar"])
t.start()

Everything works fine on my local development server. However, on cloud, every time the thread is about to finish, it gets terminated. Log said:

2014-05-12 18:45:11.826 While handling this request, the process that handled this request was found to be using too much memory and was terminated. This is likely to cause a new process to be used for the next request to your application. If you see this message frequently, you may have a memory leak in your application.

I searched a lot, people usually get this message with another message saying "reach soft memory limit" but I only got one message. I checked my memory usage report, the process is always get terminated at 40MB! but I used B1 and B2 which should have 128/256MB limit. Btw, I tried both manual and basic scaling with 1-3 instances.

Even very short run also cause this problem. I tried processing 20 images (each one only 5-10kb) should take tens of seconds to finish at maximum, it still gets killed at 40MB. I also found when my backend idel, it already cost 30 MB. I doubt the image processing will cost hundreds of MB. On my local machine, the image processing usually only take seconds to finish...

My guess is:

Since my function runs as a thread and no further requests coming, google thinks the backend is idle and should not consume such large memory. So gae kills it for memory leaking. Or, google limit the thread to use only 40MB memory.

By the way, I havn't enable my billing. Will it give me some limit like this?

Can anyone help me with this issue? Thank you very much.

Memory usage figure

Upvotes: 1

Views: 5103

Answers (1)

Salil Agarwal
Salil Agarwal

Reputation: 545

Had similar problem with backend instances while running long jobs. Clearing context cache solves the problem. Try clearing context cache:

context = ndb.get_context()
context.clear_cache()

Check this out: https://cloud.google.com/appengine/docs/python/ndb/cache#incontext

Upvotes: 2

Related Questions