CODEWITHSUNDEEP
google-cloud-platformgoogle-cloud-dataflow
Kassem Shehady
Kassem Shehady

Reputation: 760

Scheduling Dataflow pipelines

I want to schedule a google dataflow job to run every one hour

I check this url https://cloud.google.com/blog/big-data/2016/04/scheduling-dataflow-pipelines-using-app-engine-cron-service-or-cloud-functions but I got many errors.

How can I achieve this?

Upvotes: 2

Views: 1507

Answers (3)

Kassem Shehady
Kassem Shehady

Reputation: 760

You can Use a Cloud scheduler that runs every 1 hour and calls a cloud function, The Cloud function will use the Dataflow client API library to submit a Dataflow job.

Check this link https://dzone.com/articles/triggering-dataflow-pipelines-with-cloud-functions

Upvotes: 0

Kassem Shehady
Kassem Shehady

Reputation: 760

Please check doing such case using google dataflow windowing with unbounded source https://cloud.google.com/dataflow/model/windowing https://cloud.google.com/dataflow/examples/gaming-example

Upvotes: 1

Evgeny Minkevich
Evgeny Minkevich

Reputation: 2459

From my perspective, using app engine is trying to repurpose a good tool for something different.

We opted to run our own CRON instance.

Upvotes: 1

Related Questions