CODEWITHSUNDEEP
mysqlgoogle-cloud-platformcron
Steve Geggie
Steve Geggie

Reputation: 51

Best way to schedule a NodeJS process on GCP that takes more than 10 minutes

I have a nightly batch load job (14.7 million records)into a mySQL table, we have been using the GCP cron, but it occasionally times out. Do you have any suggestions or best practice tips for this type of scheduled process? We are using the following sql statements:

Our time is greatly reduced if we use a truncate statement rather than a delete, however with truncate we cannot rollback if there is a problem with the load.

Upvotes: 0

Views: 271

Answers (1)

guillaume blaquiere
guillaume blaquiere

Reputation: 75735

2 parts in your question:

  • firstly, cloud scheduler can handle task up to 30 minutes. Therefore, if you use Cloud Run to run your script, you are no longer limited to 9 minutes as you are with Cloud Functions (if my assumption is the right one)
  • Secondly, I can recommend, if it's possible in your database, to start by loading the data into a temporary table. if the load is OK, truncate your table and insert select the data from the temporary table to the target table.

Upvotes: 1

Related Questions