CODEWITHSUNDEEP
pythongoogle-bigquerygoogle-cloud-storage
user1367204
user1367204

Reputation: 4797

How to load data from one project's bucket into another project's table in python?

I have two projects. I store data in project_A and build tables in project_B. I have created a service_account.json for both of them, but I don't know how to use both files at the same time when I need to load the data from project_A and build the tables in project_B.

The data is stored in URI:

gs://project_A//*

The table will live in project_B table name huge_table

from google import storage, bigquery

proj_a_client =  storage.Client.from_service_account_json(service_acct_A.json)
proj_b_client = bigquery.Client.from_service_account_json(service_acct_B.json)

dest_table = proj_b_client.dataset('DS_B').table('huge_table')
uri = 'gs://project_A//*'

job_config = bigquery.LoadJobConfig()
load_job = proj_b_client.load_table_from_uri(uri,
                                         dest_table,
                                         job_config=job_config)

But I get the error:

google.api_core.exceptions.Forbidden: 403 Access Denied: File gs://project_A/: Access Denied

Upvotes: 1

Views: 1251

Answers (1)

yan-hic
yan-hic

Reputation: 1544

You have to make sure service_acct_B has storage access to project_A:

In project_A,

  • go to IAM & admin
  • add member service_acct_B with (at a minimum) Storage Object Viewer role

As a matter of fact, you don't use/need a service_acct_A here so proj_a_client = storage.Client.from_service_account_json(service_acct_A.json) is redundant.

Upvotes: 1

Related Questions