CODEWITHSUNDEEP
google-cloud-platformgoogle-cloud-dataflow
Paul Michaels
Paul Michaels

Reputation: 16695

How do I restart a cancelled Cloud Dataflow streaming job?

I've created a standard PubSub to BigQuery dataflow. However, in order to ensure I wasn't going to run up a huge bill while offline, I cancelled the dataflow. From the GCP console, there doesn't seem to be an option to restart it - is this possible, either through the console, or through the shell (and if so, how)?

Upvotes: 13

Views: 12429

Answers (2)

Mayank Maheshwari
Mayank Maheshwari

Reputation: 303

You can restart the job immediately by cloning. You should see an option clone at the top.

Upvotes: 3

Andrew Mo
Andrew Mo

Reputation: 1463

Cloud Dataflow currently does not provide a mechanism to restart a Dataflow job that has been stopped or cancelled.

However, for this Pub/Sub -> BigQuery flow, one way to approach this would be to use the Google-provided Pub/Sub to BigQuery template; these templates provide code-free solutions for common data movement patterns using Cloud Dataflow.

You can execute a streaming Dataflow job using this template, via the REST API, using a unique job name to ensure that there is only one instance of this Dataflow job running at any point in time. If the job were cancelled, you could (re)start this streaming Dataflow job by running the same command again.

Upvotes: 15

Related Questions