CODEWITHSUNDEEP
google-cloud-platformgoogle-cloud-dataflowgoogle-iamcloud-security
Joseph Lust
Joseph Lust

Reputation: 19995

Using Google Cloud Dataflow with a Custom Service Account, Pub/Sub, and Least Privilege

I want to run Dataflow jobs with a per job dedicated custom service account.

Upon creation, the Dataflow job wants to create a new Pub/Sub subscription, on deployment, to use as the watermark tracking subscription. It has the form of <SOURCE_SUB_NAME>__df_internal<HASH> where SOURCE_SUB_NAME is the actual subscription that the Dataflow pipeline has been configured to pull data from.

My question is how this can be done under the Principle of Least Privilege using a custom service account for this specific Dataflow job. Since the job needs to create a copy of the source Pub/Sub subscription, it needs to make a new subscription on the Pub/Sub topic which feeds source subscription. However, even if I grant the job service account the roles/pubsub.subscriber OR roles/pubsub.editor on the topic in question, I still get 403 errors in the pipeline, trying to call the Subscriber.CreateSubscription API endpoint. Empirically, I found I could only get Dataflow to make the new tracking subscription if I granted roles/pubsub.editor against the entire GCP project.

Given that, how can you use PLP without making your Dataflow job a Pub/Sub Editor on the entire GCP project? Being a project wide Pub/Sub editor means your job could read from any other topic, thus giving it more potential data access than necessary for a given job.

Upvotes: 0

Views: 52

Answers (2)

guillaume blaquiere
guillaume blaquiere

Reputation: 75950

You have to use 2 custom roles.:

  • ROLE_CREATION: Create subscription, with the permission pubsub.subscriptions.create, the pubsub.subscriptions.get, pubsub.subscriptions.list, pubsub.subscriptions.update could be required, I don't know exact how work your dataflow pipeline
  • ROLE_ATTACHMENT: attach the subscription to the topic, permission pubsub.topics.attachSubscription

With this 2 created, you have to grant your Dataflow Service Account like this:

  • At the project level, grant the ROLE_CREATION custom role. Like this, Dataflow will be able to create a subscription.
  • At the topic level, grant the ROLE_ATTACHMENT custom role. Like this, Dataflow will be able to use your topic.

By doing this, your dataflow can create multiple subscription, but can attach it only on the authorized topic, not on other. No data leakage like this.

Upvotes: 0

marky
marky

Reputation: 153

The error 403 refers to the incorrect IAM permission and as for your project, my insight is make a custom role with the permission only necessary to create and manage subscription (not the roles/pubsub.editor). After that, assign that custom role at the topic level (roles/pubsub.subscriber) and this will follow the PLP and avoid granting unnecessary permission.

Upvotes: 0

Related Questions