CODEWITHSUNDEEP
pythongoogle-sheetsgoogle-drive-apigoogle-cloud-dataflowapache-beam
Trusha Sahu
Trusha Sahu

Reputation: 1

"Access Denied: BigQuery BigQuery: Permission denied while getting Drive credentials" during running dataflow

I am trying to read data from the external bigquery table (i.e. linked to google sheet) in apache beam pipeline as below and later loading into bigquery table only.

But the code getting failed with below error during reading data,

Error: "Access Denied: BigQuery BigQuery: Permission denied while getting Drive credentials."

dataflow Code:

with beam.Pipeline(options=google_cloud_options) as p:
            # Step 1: Get URL list from BigQuery table
            urls = (
                p | 'Get URLs from BQ Table' >> beam.io.ReadFromBigQuery(
                    query="""
                        SELECT cast(id as INT64) AS site_id, url
                        FROM `my-project.my_dashboard.MAP_URLS_GSHEET`
                    """,
                    use_standard_sql=True
                )
            )

Could you please help here , how to solve this issue i.e. specially when running through dataflow?

Upvotes: 0

Views: 629

Answers (1)

Bruno Volpato
Bruno Volpato

Reputation: 1428

You need to add BigQuery permissions to the service account being used on the Dataflow job / workers.

Please check https://cloud.google.com/dataflow/docs/concepts/security-and-permissions#worker-service-account for more information.

Upvotes: 0

Related Questions