CODEWITHSUNDEEP
google-cloud-platformgoogle-cloud-dataflowapache-beam
Shakir Mukkath
Shakir Mukkath

Reputation: 51

Is it possible to update existing Google cloud dataflow pipelines when using template for deployment ?

When deploying google dataflow pipeline as templates, Is it possible to update the pipeline using another version of the template?

Basically, I am looking for a combination https://cloud.google.com/dataflow/pipelines/updating-a-pipeline with https://cloud.google.com/dataflow/docs/templates/overview

Upvotes: 5

Views: 2228

Answers (2)

Bruno
Bruno

Reputation: 332

Now it's possible to update a template streaming Dataflow job.

Using the REST API, set the update parameter to true:

POST https://dataflow.googleapis.com/v1b3/projects/YOUR_PROJECT_ID/locations/LOCATION/templates:launch?gcsPath=gs://YOUR_BUCKET_NAME/templates/TemplateName
{
    "jobName": "JOB_NAME",
    "parameters": {
        "topic": "projects/YOUR_PROJECT_ID/topics/YOUR_TOPIC_NAME",
        "table": "YOUR_PROJECT_ID:YOUR_DATASET.YOUR_TABLE_NAME"
    },
    "environment": {
        "tempLocation": "gs://YOUR_BUCKET_NAME/temp",
        "zone": "us-central1-f"
    }
    "update": true
}

The update option is not present using gcloud dataflow jobs run.

https://cloud.google.com/dataflow/docs/guides/templates/running-templates#example-3:-updating-a-custom-template-streaming-job https://cloud.google.com/dataflow/docs/reference/rest/v1b3/LaunchTemplateParameters

Upvotes: 2

Zhou Yunqing
Zhou Yunqing

Reputation: 444

The feature of updating an existing job from template API is not ready yet. (We are working on it).

At the time, you can probably make use our public repo (basically the source code for these templates) to do it. Basically, you can just build and launch the job to "update" the running job from a shell.

https://github.com/GoogleCloudPlatform/DataflowTemplates

Upvotes: 3

Related Questions