CODEWITHSUNDEEP
google-cloud-platformgoogle-cloud-dataflow
prideloki
prideloki

Reputation: 491

Dataflow worker subnetwork is not accessible to Dataflow Service account

When i run a job on Dataflow, i get an error.

INFO: 2018-05-13T04:43:38.642Z: Checking required Cloud APIs are enabled.
May 13, 2018 11:44:09 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-13T04:43:38.927Z: Checking permissions granted to controller Service Account.
May 13, 2018 11:44:09 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
SEVERE: 2018-05-13T04:43:51.378Z: Workflow failed. Causes: Subnetwork regions/asia-southeast1/subnetworks/SUBNETWORK is not accessible to Dataflow Service account
May 13, 2018 11:44:09 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-13T04:43:53.038Z: Cleaning up.

My command line

mvn compile -X exec:java -Dexec.mainClass=com.to.class \
"-Dexec.args=--runner=DataflowRunner \
...
--network=NETWORKNAME \
--subnetwork=regions/asia-southeast1/subnetworks/SUBNETWORK \
--zone=asia-east1-a \
--region=asia-east1 \
" -Pdataflow-runner

Upvotes: 2

Views: 2621

Answers (1)

Guillem Xercavins
Guillem Xercavins

Reputation: 7058

The subnetwork is in the asia-southeast1 region whereas the workers are in asia-east1. You can use a subnetwork in asia-east1 or provision the VMs in one of the asia-southeast1 zones instead but both will need to match. For the latter approach, take into account that asia-southeast1is not a supported regional endpoint as of now so you can't specify it with the --region flag for control and metadata purposes (you can however spin up workers with the --zone flag).

Upvotes: 3

Related Questions