CODEWITHSUNDEEP
pythongoogle-cloud-platformgoogle-cloud-dataflow
Simon Hawe
Simon Hawe

Reputation: 4539

How to set region when running a dataflow flex template using FlexTemplatesServiceClient

I am trying to launch a dataflow job using the python FlexTemplatesServiceClient. I am successful with creating a job. However, my problem is that the jobs always ends up in region us-central1 no matter what I do. What I have tried so far is

flex_client = FlexTemplatesServiceClient.from_service_account_file(auth_info)
p = {
    "project_id": "joyn-data-dev",
    "location": "europe-west4",
    "launch_parameter": {
        "job_name": "some-name",
        "container_spec_gcs_path": "gs://some-bucket/flex-template-spec.json",
        "environment" : { # Tried using this, and not using this. nothing had an effect
            "zone" : "europe-west4",
            #"worker_zone": "europe-west4",
            #"worker_region": "europe-west4",
        }
    },
}
response = flex_client.launch_flex_template(p)

nothing of that had an effect so far. When I launch the exact same job using the rest API or the glcoud cli, everything works fine. So for example the following command

gcloud dataflow flex-template run "somename-`date +%Y%m%d-%H%M%S`" \
    --template-file-gcs-location "gs://some-bucket/flex-template-spec.json" \
    --region europe-west4

works fine without any issue.

Unfortunately, the client API documentation is rather sparse and I couldn't find any help there. Hoping someone face the similar problem and solved it.

Upvotes: 0

Views: 423

Answers (1)

Simon Hawe
Simon Hawe

Reputation: 4539

Apparently, there is a bug in the GRPY implementation of that API that ignores the region/location configuration you specify. Currently, you have to use the rest API if you want to trigger a dataflow programmatically.

Upvotes: 2

Related Questions