CODEWITHSUNDEEP
google-cloud-platformgoogle-bigquery
Indrajeet Gour
Indrajeet Gour

Reputation: 4490

how to update Big Query back-end data on each upload for bucket

I have created the Big Query out of the data I have in my Cloud storage bucket.

In my use case, I am sending data periodically to the same bucket which is backend of my Big Query(while creating the Big query table I used the same bucket name).

Is it possible to get the updated data into Big Query, as I am pushing new data each time into the same bucket on some interval basis.

Just to mention - I am making native Big query from my dedicated storage bucket mentioned above.

Your help will be much appreciated. thanks in advance.

Upvotes: 0

Views: 1726

Answers (1)

Alexey Maloletkin
Alexey Maloletkin

Reputation: 1099

You can create an external (federated) table on Google Cloud Storage Bucket. In this case, whenever you query this table you will get the latest data.

If you just need to append data to a table (let's call it target table) based on data from the bucket - I can imagine following this process:

  1. Create a federated table on the GCS bucket
  2. Setup a simple cron job that runs a bq command which is just doing select * from [federated_table] and appends results into the target table (you may have a more complicated query that will check duplication of data in the target table and only appends new data).

Alternative option: Setup a trigger on your bucket that activates cloud function and in a cloud function you just load the newly added data to the target table.

Upvotes: 4

Related Questions