In GCP, is there a canonical way to scrape data from an API?

Question

I'm building an application that will periodically pull data from several APIs and write them to cloud storage for later processing by Dataflow. There are many different ways to do this so I wanted to sanity check before I jumped in.

My plan is this:

• For each API, Cloud Scheduler will hit an endpoint for an App Engine app
• The app will create a Compute Engine VM instance with a startup script that pulls the data from the API and writes it to storage
• When done, the VM will hit another endpoint on the App Engine app that shuts down the VM.

Is this a reasonable way to perform this sort of action? Is there a simpler or more straight-forward method? Thank you in advance for the replies.

Answer 1

Cloud Scheduler can schedule Compute Engine without App Engine however it seems that you cannot create and delete the VM with this method.

Answer 2

You can just use App Engine cron jobs to schedule the tasks. Your App Engine app cron handler can simply run the script that pulls data from the APIs. Maybe I am missing something, why do you need to use a Compute Engine instance to run the script?