CODEWITHSUNDEEP
google-cloud-dataflow
Raj
Raj

Reputation: 11

How to config dataflow Pipeline to use a Shared VPC?

I know we have configuration arguments where you can specify network and subnets, I tried doing that but with a Shared VPC network, it gives me this error.

enter image description here

Upvotes: 0

Views: 1593

Answers (2)

Armin_SC
Armin_SC

Reputation: 2260

The usage of subnetworks in Cloud Dataflow require to specify the subnetwork parameter when running the pipeline; However, in the case of subnetwork that are located in a Shared VPC network, it is required to use the complete URL based on the following format:

https://www.googleapis.com/compute/v1/projects/<HOST_PROJECT>/regions/<REGION>/subnetworks/<SUBNETWORK>

Additionally, verify you are adding the project's Dataflow service account into the Shared VPC's project IAM table and give it the "Compute Network User" role permission in order to ensure that the service has the required access scope.

You can take a look on the Subnetwork parameter official Google's documentation which contains detailed information about this matter.

Upvotes: 2

Guillem Xercavins
Guillem Xercavins

Reputation: 7058

Be sure to include the Project ID in the --subnetwork option:

/projects/<PROJECT_ID>/regions/<REGION>/subnetworks/<SUBNETWORK>

and give to the Dataflow Service account the Network User role in the host project, which is what I suspect is going on according to the error message.

Upvotes: 0

Related Questions