CODEWITHSUNDEEP
google-cloud-platformgoogle-bigquery
Vibhor Jain
Vibhor Jain

Reputation: 1476

Streaming data to BQ - ideal least privilege needed for the Service Account

Streaming data to BQ - https://cloud.google.com/bigquery/docs/reference/rest/v2/tabledata/insertAll

May I ask why BigQuery Metadata Viewer access is needed on the Dataset? (despite destination tale already exists)

With just BigQuery Data Editor at Table level, it gives the following error: google.api_core.exceptions.Forbidden: 403 POST https://bigquery.googleapis.com/bigquery/v2/projects/PROJECT/datasets/DATASET/tables/TABLE/insertAll: Access Denied: Dataset PROJECT:DATASET: User does not have bigquery.datasets.get permission for dataset PROJECT:DATASET.

===== UPDATE:

table_ref = bigquery.DatasetReference(project = BQ_PROJECT, dataset_id = BQ_DATASET).table(table_id = BQ_TABLE)
table =  client.get_table(table_ref)

rows_to_insert = [
    {"col1": "streaming-r1", "col2": "A", "col3": "3"},
    {"col1": "streaming-r2-", "col2": "B", "col4": 1},
]


stream_resp = client.insert_rows(table_ref,rows_to_insert,
                                 selected_fields=table.schema)

Upvotes: 0

Views: 329

Answers (1)

rodvictor
rodvictor

Reputation: 349

I have created the following Feature Request in Google Cloud Platform (1) about your suggestion of being able to insert data into a table without dataset permissions.

This can be helpful when the Project owner does not want to give everyone dataset access for security reasons. Instead, it would be useful to be thrown an exception if the table does not exist without needing 'bigquery.datasets.get' permission.

Make sure you upvote it (2) so it is more likely to be implemented. You can also subscribe (3) to the feature request so you can get notified every time there is a new message on the thread.

Upvotes: 0

Related Questions