CODEWITHSUNDEEP
pythongoogle-cloud-platformgoogle-cloud-dataflowapache-beampublish-subscribe
Martin Wiebusch
Martin Wiebusch

Reputation: 1803

Dataflow: Stream pub/sub messages from different project

I have a pub/sub topic in project A. I would now like to stream messages from that topic into a dataflow pipeline running in a different project B. I have followed the example at https://cloud.google.com/pubsub/docs/stream-messages-dataflow and everything works when the topic is in the same project as the dataflow pipeline. However, when trying to stream messages from a topic in a different project I get the following permissions error:

INFO:apache_beam.runners.dataflow.dataflow_runner:2022-11-09T16:34:40.349Z: JOB_MESSAGE_ERROR: Workflow failed. Causes: Check if topic projects/XXXXXXX/topics/test-topic exists failed with error: User not authorized to perform this action.

The service account which runs the pipeline has the Pub/Sub Admin role in both projects. I even tried making it Owner in project A (where the topic lives), but no success. I always get the same error.

Upvotes: 0

Views: 609

Answers (2)

Mazlum Tosun
Mazlum Tosun

Reputation: 6572

To solve this issue :

  • Run the Dataflow job in the project B
  • The service account is created for example in the project B
  • In the IAM menu of projet A, give the permissions to read from the Pub Sub in the projet A
  • In the IAM menu of projet B, give the permissions to launch a Dataflow job in the project B
  • When you launch your Dataflow job in the project B, set a program argument indicating what Service Account email launches the job :

For Dataflow Python :

--service_account_email=my-service-account-name@<project-id>.iam.gserviceaccount.com

For Dataflow Java :

--serviceAccount=my-service-account-name@<project-id>.iam.gserviceaccount.com

Upvotes: 0

al-dann
al-dann

Reputation: 2725

You might need to create a subscription in the source project (A), so your dataflow job (in the project B) takes the messages from that subscription (from the project A).

Then yoou find out a service account under which your dataflow job is runnig (in the project B). Presumably that service account is in the project B. And provide relevant permissions to that service account, so it can work with the source subscription (from the project A).

Upvotes: 1

Related Questions