CODEWITHSUNDEEP
google-cloud-platformgoogle-bigquerygoogle-cloud-dataflow
John Smith
John Smith

Reputation: 347

Google Analytics -> Bigquery -> daily export to IBM cloud storage

We have enabled GA 360 export to Bigquery that creates a new table everyday. For further processing we need the daily export to be transferred to IBM cloud object storage.

The easiest solution would be to use CLI -

a) bigquery export to Google cloud storage 

bq extract

and then

b) move the extract to IBM object storage

gsutil cp

Couple of questions in this regard -

1) Can we skip the google cloud storage in between and is it possible to export directly to IBM bucket ?

2) We have not used Data flows before. Can this use case be a candidate for using Data flows ? Or what can be the easiest solution natively within GCP ?

Upvotes: 2

Views: 259

Answers (1)

Pentium10
Pentium10

Reputation: 207912

The best option you have here, is to use the bq extract in a scheduled manner for that you can use Cloud Scheduler assembled in a Cloud Functions.

  1. Cloud Scheduler hits a Cloud Functions endpoint.
  2. GCF runs the extract job to GCS
  3. Another GCF observes the new file in a GCS bucket and notifies/uploads to IBM Cloud Storage.

It took us less than an hour to wire it up and get it working. As you have to setup essentially two cloud functions, and the scheduler.

Later this year, you will have a way to detect via Functions when a BigQuery table was created, so you could take out the scheduler and immediately as the table has been created, you can export it.

As an additional step you could delete the exported GCS file using another GCF triggered by http when you completed the move. And essentially you could also move the data into a partition to take advantage of the long term storage reduced pricing.

Upvotes: 2

Related Questions