CODEWITHSUNDEEP
apache-sparkgoogle-bigquerygoogle-cloud-dataproc
TheRealJimShady
TheRealJimShady

Reputation: 4293

DataProc BigQuery Connector Access Across Projects

I am writing a Spark Job to run on a DataProc cluster in Project A but the job itself will pull data from a BigQuery instance in Project B using the BigQuery Connector. I have owner privileges for both project, but the job is run using a service account. The response I'm getting in the stack trace is this:

{
  "code" : 403,
  "errors" : [ {
    "domain" : "global",
    "message" : "Access Denied: Table ABC:DEF.ghi: The user [email protected] does not have bigquery.tables.get permission for table ABC:DEF.ghi.",
    "reason" : "accessDenied"
  } ],
  "message" : "Access Denied: Table ABC:DEF.ghi: The user [email protected] does not have bigquery.tables.get permission for table ABC:DEF.ghi."
}

Upvotes: 1

Views: 1331

Answers (1)

Dennis Huo
Dennis Huo

Reputation: 10677

As you noticed, Dataproc clusters run on behalf of service accounts instead of individual users; this is intentional, since different users may be creating Dataproc clusters in a shared project where they do not want their personal permissions to leak across other members of the org using the same project, and instead should define permissions according to service accounts which each represent a particular scope of workloads.

In this case, all you have to do is go into project B and add the service account from project A as one of the roles that can access BQ in project B. If it's not a complex arrangement withlots of users and different teams, you could just add it as "project viewer" on project B, otherwise you'll want something more fine-grained like a "bigquery viewer" or "bigquery editor".

Add that service account the same way you would add any user to project B.

Upvotes: 4

Related Questions