CODEWITHSUNDEEP
google-cloud-platformgoogle-cloud-functions
avemike
avemike

Reputation: 117

Google Cloud Function timeout overrun workaround/alternative

My function needs to run once every day. I thought of using Google Cloud Functions service along with Cloud Scheduler(to automate running it) in that case. The problem is Cloud Functions have maximum timeout set as 9 minutes (after that time function is terminated). My function isn't CPU intensive or memory intensive, but needs around ~30 minutes to execute. Are there any workarounds/alternatives for that case?

Little bit of context: the function I want to run once per day is simple data synchronization between two REST APIs. It takes so long, because of low rate limit of one API.

Upvotes: 2

Views: 2180

Answers (3)

Ocean Overflow
Ocean Overflow

Reputation: 571

You can use Cloud Functions (2nd gen) now. With longer request workloads up to 60 minutes.

Upvotes: 1

guillaume blaquiere
guillaume blaquiere

Reputation: 76073

An alternative is to use Cloud Run (you only need to add a webserver and to build a standard container. If you share your language, I can provide you examples). Cloud Run has the same behavior as Cloud Function (if you set the concurrency to 1) and can handle request up to 60 minutes.

However, be careful of Cloud Scheduler. It is able to handle request only up to 30 minutes, after, it considers the request as failed because of timeout. But, the Cloud Run request can continue correctly up to 60 minutes (after Cloud Run stops the processing). In this case, don't set up retry policy on your Cloud Scheduler and let the Cloud Run continue its processing a full hour. The bad side of that is that you can't use Cloud Scheduler retry policy in case of real error.

Upvotes: 3

samthecodingman
samthecodingman

Reputation: 26296

Because you know the rate-limit of the API, you should be able to calculate how many calls you can do in an 8 minute period. Then after you've made those calls, save your progress to Cloud Firestore in a location monitored by another Cloud Function listening to it using onCreate, such as /_functionsAsyncTasks. In this Cloud Function, pick up where you left off and do another 8 minutes worth of calls. Repeat until you've worked through all your calls.

Upvotes: 2

Related Questions