GCP Cloud function is getting timeout

Question

I have created a web scraping python script and its running fine on my local system it takes 30 mins.

But when I tried to put the script on GCP cloud function it threw timeout after 60004 ms.

2022-03-16T11:41:01.420461007Zget_training_databudock8slftb Function execution took 60004 ms, finished with status: 'timeout'
Function execution took 60004 ms, finished with status: 'timeout'

To complete this I used the following services.

Cloud scheduler -> Pub/Sub -> cloud function

Could you please suggest which GCP service should I pick to run the python script with cost-effective and which runs daily?

Answer 1

You can now use 2nd generation Cloud Functions.

If you create a 2nd gen Cloud Function and use an https trigger, you can set the max timeout to 3600 seconds = 1 hour instead of 10 minutes.

For info on 2nd generation Cloud Functions:
https://cloud.google.com/functions/docs/2nd-gen/overview

See also here on how to combine with Cloud Scheduler:
Do Google Cloud background functions have max timeout?

Answer 2

Function execution time is limited by the timeout duration, which you can specify at function deployment time. By default, a function times out after 1 minute or 60000ms which you're experiencing, but you can extend this period up to a maximum of 9 minutes.

You can extend it by editing your deployed function and set the timeout to 540 seconds.

For more information, you may also refer on this documentation.

---

If your scraper took 30mins on your local, then maybe you need to optimize it first and logically set a timer before 9mins and create another schedule by using Google Cloud Client Libraries for google-cloud-scheduler to logically continue where you end and start on it until it finishes the scraping.