CODEWITHSUNDEEP
google-cloud-platformgoogle-bigqueryjob-scheduling
Siddhant Naik
Siddhant Naik

Reputation: 51

How to schedule a query (Export Data) from Google Big Query to External Storage space (Eg: Box)

I read many articles and solutions regarding scheduling queries to external storage places in Google Big Query but they didn't seem to be that clear.

Note: My company has subscription only to Google Big Query and not to the complete cloud Services (Google Cloud Platform).

I know how to do it manually but I am looking to automate the process since I need the same data every week.

Any suggestions will be appreciated. Thank you.

Upvotes: 0

Views: 2036

Answers (1)

Tamir Klein
Tamir Klein

Reputation: 3632

Option 1

You can use Apache Airflow which provides the option to create schedule task on to of BigQuery using BigQuery operator.

enter image description here

You can find in this link the basic steps required to start setting this up

option 2

You can use the Google BigQuery command line to export your data as you do from the webUI, for example:

bq --location=[LOCATION] extract --destination_format [FORMAT] --compression [COMPRESSION_TYPE] --field_delimiter [DELIMITER] --print_header [BOOLEAN] [PROJECT_ID]:[DATASET].[TABLE] gs://[BUCKET]/[FILENAME]

Once you get this working you can use any schedule process of your liking to schedule the run of this job

BTW: Airflow has a connector which enables you to run the command line tool

Once the file in GCP you can use Box G suite integration to see and manage your files

enter image description here

Upvotes: 3

Related Questions