Error creating dataflow pipeline using custom template on GCP

I am creating dataflow pipeline for Custom Apache Beam Template from GCP console for custom template I already uploaded to Google Cloud Storage.

1 After selecting template path, I am seeing this warning or may be error

Template metadata error details
Fail to process as Flex Template and Legacy Template. Flex Template Process result:No SDK language information is provided., Legacy Template Process result:(35197769719996987970000d1381234f): Unable to open template file: gs://abc/extraction/templates/cron-template_metadata.

I do have gs://abc/extraction/templates/cron-template because that's what i specified in my args --templateLocation=gs://abc/extraction/templates/cron-template file but not gs://abc/extraction/templates/cron-template_metadata.

I am confused if it is a warning or error.

2 After I hit button for creating the pipeline

I am getting this error

Job creation failed
User cannot act as Dataflow worker service account

Request ID: 38414979878078088000807058

What permission do I need in order to create a pipeline. I have Dataflow Admin permission.

Upvotes: 0

Answers (2)

XQ Hu

Reputation: 407

More about the permissions for Flex Templates: https://cloud.google.com/dataflow/docs/guides/templates/configuring-flex-templates#permissions

Upvotes: 0

SRJ

Reputation: 2846

1 Template metadata error details

Fail to process as Flex Template and Legacy Template. Flex Template Process result:No SDK language information is provided., Legacy Template Process result:(35197769719996987970000d1381234f): Unable to open template file: gs://abc/extraction/templates/cron-template_metadata.

This is an warning and can be ignored.

2 User cannot act as Dataflow worker service account

We need to add a below permission to user who is creating the dataflow pipeline and this can be fixed

Service Account User

Users granted the Service Account User role on a service account can use it to indirectly access all the resources to which the service account has access. For example, if a service account has been granted the Compute Admin role (roles/compute.admin), a user that has been granted the Service Account Users role (roles/iam.serviceAccountUser) on that service account can act as the service account to start a Compute Engine instance. In this flow, the user impersonates the service account to perform any tasks using its granted roles and permissions.

Reference : https://cloud.google.com/iam/docs/service-account-permissions#user-role

Upvotes: 1