CODEWITHSUNDEEP
automationgoogle-bigquery
Brian Doherty
Brian Doherty

Reputation: 97

Automating Google BigQuery query without local access to APIs

I'm trying to automatically run a bigquery query every week and save the data onto my computer.

Ideally this would be done through the APIs but due to issues with proxies and firewalls in work I cannot access the APIs on my computer.

Are there any other options available to me? I have tried looking into Google Compute Engine but I'm not quite sure if this will help me

It is necessary to get the data from BigQuery as well, standard GAP reporting through the WebUI will not suffice. And the query will result in ~ 2,000,000 rows of data each week

Upvotes: 2

Views: 1258

Answers (1)

Mikhail Berlyant
Mikhail Berlyant

Reputation: 173046

As you might know, BigQuery by itself doesn’t provide any scheduling mechanism.
You should use 3rd party solution or develop your own.
As an example, you can use App Engine Cron Service to automate BQ query execution and exporting result to GCS.
https://cloud.google.com/appengine/docs/java/config/cron https://cloud.google.com/appengine/features/#cron https://cloud.google.com/bigquery/bigquery-api-quickstart

To get your result from GCS to local computer you still need some sort of access to cloud and use respective tool (see link below)
https://cloud.google.com/storage/docs/gsutil

Upvotes: 2

Related Questions