CODEWITHSUNDEEP
google-bigquerydatasetgoogle-alerts
Maciej Jankowski
Maciej Jankowski

Reputation: 25

BigQuery Dataset - Uploaded rows with Scheduled query

I have a scheduled query running every 24 hours that is loading some data into BQ table (this data is about some problems from the previous day). Additionally, I want to have an alert whenever the daily partition of that table is not empty. For this case I created an alert with BigQuery Dataset - Uploaded rows metric. Unfortunately, this metric behaves as if no data was added to my dataset. Does it mean that data added by a scheduled query is not counted there? If yes - how can I create an email alert (not to myself, to some other email) a BQ daily partition of a table contains any rows?

Upvotes: 0

Views: 77

Answers (1)

Pratik Patil
Pratik Patil

Reputation: 832

Scheduled or not, the data will be counted towards all metrics.

In GBQ the load jobs are "batch" jobs. One can insert as much data as possible but it will be processed when resources are available in the GBQ backend. It's possible that sometimes a user creates a job and you have to wait if the worker pool is resizing or full or something along those lines.

For the same reasons, your data may not be immediately accessible or some metric may not have updated in time. The larger the data you are trying to add in, the longer it may take to show up.

Usually, I work with very large amounts of data, so as a standard practice I also add a few minutes of sleep timer before I want to access that data again. In addition you can also check endTime of the job to see if it's completely done or not.

As far as custom email alerts or checking some preconditions like if a partition is empty etc are concerned you have to manage those by yourself in your program. (if you are not using any GBQ workflow platforms like Magnus (https://potens.io/products/#magnus))

Upvotes: 1

Related Questions