CODEWITHSUNDEEP
google-cloud-platformgoogle-bigquery
user7665394
user7665394

Reputation: 39

GCS to BigQuery table using stored Proc

I want to create a stored Proc which can read data from GCS bucket and store into the table in bigquery. I was able to do it using python bu conecting to gcs and creating bigquery client.

credentials = service_account.Credentials.from_service_account_file(path_to_key)
bq_client = bigquery.Client(credentials=credentials, project=project_id)

Can we achieve the same using stored procedure?

Upvotes: 0

Views: 773

Answers (1)

Roman Czerwinski
Roman Czerwinski

Reputation: 559

Have you looked into using an external table? You can query directly off of Google Cloud Storage without needing to load anything. Just define the schema of the expected data along with the GCS URI's and once the data is in GCS it is accessible via SQL in BigQuery. Otherwise, no. There is no LOAD statement that you can execute via BigQuery SQL. See the docs here for all of the ways to load data into a table. You could have the external table and maybe create a stored procedure that does an INSERT into another table using the data from the external table you created. That is if you are really hell bent on having a stored procedure to "load" data to a normal BigQuery table. Otherwise the external tables are an excellent option to obviate the need to even load the data in the first place.

Upvotes: 1

Related Questions