CODEWITHSUNDEEP
google-cloud-platformgoogle-cloud-functions
mikez
mikez

Reputation: 45

Google Cloud Function - Memory Limit Exceeded (2GB) - Heavy data processing

I have a heavy data processing script written in Python. Each time the script processes one job, about 500MB of RAM is used. (The reason is because the script looks up historical records from a very large database.) The processing script also takes about 3 minutes for each row to run.

We have deployed our python script to a Google Cloud Function. When we invoke the function to process three jobs simultaneously, the function works fine, memory usage is about 1500-1600MB; all is dandy.

However, when we try to invoke the function to process 10 jobs or 100 jobs simultaneously, the function is killed as memory is exceeded. We noticed in the documentation that the memory limit for a function at any one time is 2GB. Would it be safe to say that we can't increase that to 10GB or 100GB or 1000GB so we can run more instances of the script in parallel? To be honest, why is it 2GB per function, not 2GB per invocation? I would love to have access to serverless capabilities for heavy data processing work on Google; this does not seem to be available.

If so, would you say that the best way to achieve our goal is just use a stock-standard Google VM with 1000GB of RAM?

Thanks.

Upvotes: 3

Views: 3520

Answers (1)

guillaume blaquiere
guillaume blaquiere

Reputation: 76073

The 2Gb is per instance. When a function is trigger, an instance is spawned. If the function is not used, after a while (10 minutes, more or less, without commitment), the instance end.

However, if there is a new request and an instance is up, the existing instance is reused. And, if there is a lot of request, new instances are spawned. An instance of function can handle only 1 request in same time (no concurrency)

So, when your instance is reused, all the element in your execution environment is reused. If you don't cleanup the memory and/or the local storage (/tmp which is an in memory storage), you have a memory leak and your function crash.

Take care of your memory and object handle, clean well your context. If your request can handle 1 job, it must be able to handle 10 or 100 successive jobs, without crash.

UPDATE

I'm not Python expert but for cleaning the memory I use this

import gc
gc.collect()

Upvotes: 9

Related Questions