CODEWITHSUNDEEP
google-app-enginememory-leakspython-2.7
fspeech
fspeech

Reputation: 75

Why is memory not released after function returns?

On GAE my handler calls a function that does all the heavy lifting. All the objects are created within the function. However after the function exits (which returns a string for response.out.write) the memory usage does not go down. The first http call to GAE works, but memory stays at about 100MBs afterwards. The second access attempt fails because private memory limit is reached.

I have cleared all class static objects that I wrote and called the close and clear functions of the third party library to no avail. How does one cleanly release memory? I'd rather force a restart than tracking down memory leaks. Performance is not an issue here.

I know that it is not due to GC. GAE reports that memory stays at high level for a long period of time. The two http calls above were separated by minutes or longer.

I've tried to do the import of my function in the Handler.get function. After serving the page I tried to del all imported third party modules and then my own module. In theory now each call should get a restart of all suspected modules but memory problem still persists. The only (intended) modules left between calls should be standard library modules (including lxml, xml etc).

EDIT: I now use taskqueue to schedule the heavy duty part on a backend instance and use db.Blob to pass around the results. Getting backends to work solves the memory issue. GAE documentation on backends is complete but confusing. The key is that one needs to follow the instructions on 1) editing backends.yaml 2) using appcfg to update (deploying from launcher is not enough). Afterwards check in admin that the backends is up. Also taskqueue target= breaks on the development server so one needs to work around it on the development server.

Upvotes: 2

Views: 570

Answers (1)

Filip Roséen
Filip Roséen

Reputation: 63852

This is (probably) due to the fact that there is nothing saying that the garbage collector (which is in charge of freeing unused memory) will kick in directly when your function returns.

You could manually force it to kick in via a few hacks but that will not solve anything if two http request happens aprox. at the same time.

Instead I recommend you to look over solutions which doesn't require you to do the heavy lifting on each request.

If the data generated is unique for each request see if you can do the computations outside of your (limited) private memory pool.

how do I manually start the Garbage Collector?

When your heavy weight variables have gone out of scope invoke the GC by using the below method.

import gc

...

gc.collect ()

Upvotes: 2

Related Questions