CODEWITHSUNDEEP
google-bigquerygoogle-cloud-storageetl
user2550621
user2550621

Reputation: 77

Loading data from Bigquery to google storage bucket in CSV file format

I run a dataset in bigquery on a daily basis which i need to export to my google storage bucket. The dataset is greater than 10MB which means i'm unable to use app-scripts.

Essentially, I'd like to automate a data load using my bigquery script which exports the dataset as a CSV file to google storage.

Can anyone point me into the right direction in terms of which programme/method to use. Please also share your experiences.

Thanks

Upvotes: 3

Views: 1528

Answers (1)

pgiecek
pgiecek

Reputation: 8200

Here you can find some details on how to export data from BigQuery to Cloud Storage along with a sample written in Python.

https://cloud.google.com/bigquery/exporting-data-from-bigquery

You can implement a simple application running on App Engine that will contain cron job scheduled to run once a day and perform the steps described in the tutorial above.

https://cloud.google.com/appengine/docs/python/config/cron

Upvotes: 3

Related Questions