CODEWITHSUNDEEP
google-bigquery
sanket singh
sanket singh

Reputation: 11

How to append every day data to the tabels in the big query?

We are migrating from Digital Ocean to GCP and to test things out we exported our data as json from our mongo DB and uploaded it to GCS bucket (userDump.json is one of it)

Now, we are fetching data from our GCS bucket and making tables in big query. (users table)

So far everything is working out.

My problem:

Every day we are onboarding new users and their data is saved on GCS. We want to run a cron /similar functionality to add that data to the table so in the morning, people can perform queries on yesterday's data.

How can I achieve this?

Upvotes: 1

Views: 983

Answers (1)

Felipe Hoffa
Felipe Hoffa

Reputation: 59225

Take a look at my lazy data loading in BigQuery article:

What you could do is:

  • Have BigQuery read files directly from GCS - a federated query.
  • Then have a scheduled query inside BigQuery materialize these federated tables into native BigQuery tables.
  • Your users will get fresh data daily, or even more frequently - no servers needed :).

Upvotes: 2

Related Questions