CODEWITHSUNDEEP
google-cloud-platformgoogle-cloud-dataflow
Tarique
Tarique

Reputation: 711

Dataflow Job failing

I have a pipeline which requires a Dataflow Job to run. I was using the gcloud CLI command to start a dataflow job which was working fine for over a month. But since last three days the dataflow job is failing within 10-20 sec with the following error log.

Failed to start the VM, launcher-2022012621245117717885921401920990, used for launching because of status code: UNAVAILABLE, reason: One or more operations had an error: 'operation-1643261093401-5d68989bed339-a33de830-9f90d92a': [UNAVAILABLE] 'HTTP_503'..

The command I'm using is:

gcloud dataflow sql query "SELECT tr.* FROM pubsub.topic.`my_project`.pubsub_topic as tr" 
    --job-name test_job
    --region asia-south1
    --bigquery-write-disposition write-empty
    --bigquery-project my_project
    --bigquery-dataset test_dataset --bigquery-table table_name
    --max-workers 1 --worker-machine-type n1-standard-1

I tried starting the job from cloud console with same parameters as well which failed with the same error log. I have tested the job run from console before and it worked fine. The issue started a couple days ago.

What could be going wrong?

Thanks.

Upvotes: 0

Views: 1085

Answers (1)

Tim Chiang
Tim Chiang

Reputation: 394

The Google Cloud error model indicates that a 503 means the service is unavailable [1].

You may try to change the region, for example, from europe-north1 to europe-west4, that should work. Additionally, you shouldn't include your job ID on Stack Overflow.

[1] https://cloud.google.com/apis/design/errors#handling_errors

Upvotes: 1

Related Questions