CODEWITHSUNDEEP
google-cloud-functionsgoogle-cloud-dataflowdataflow
Roshan Fernando
Roshan Fernando

Reputation: 525

Cloud functions to schedule Dataflow effectively?

What are some best options to schedule Dataflow job using Cloud functions? I would want to schedule the dataflow job to start and stop at some time.

The dataflow job might be reading from an unbounded source and might run continuously, so looking to have it scheduled to stop.

Upvotes: 1

Views: 494

Answers (1)

Temu
Temu

Reputation: 879

Since Google Cloud Functions enable you to write functions to respond to events in your cloud environment, regarding to stop Dataflow jobs I would use a cron job.

It will just send a request to a defined URL in your app which calling the API can do whatever you consider.

cron:
- description: "start dataflow job"
  url: /tasks/summary
  schedule: every 24 hours
- description: "stop dataflowjob"
  url: /tasks/stop
  schedule: every day 20:00
  timezone: Australia/NSW
- description: "monday morning summary"
  url: /mail/weekly
  schedule: every monday 09:00
  timezone: Australia/NSW

You would need anyway to do a HTTP request in order to trigger the Cloud Function.

Upvotes: 0

Related Questions