CODEWITHSUNDEEP
google-bigquery
Ck87
Ck87

Reputation: 41

How do I get usage data about the what views and datasets are being used/queried in BigQuery?

I need usage data on the dataset and views in BigQuery. I want a count of how many queries were ran against all the different datasets and views. My goal is to understand what datasets and views are currently being used.

I looked in the Audit Log, however I'm not able to export all the records into excel for analysis. I am only able to export max 300 logs for a specific day. Is there a way to run a query in BigQuery that will give me this data instead of exporting from the Audit Log?

Upvotes: 1

Views: 1885

Answers (1)

justbeez
justbeez

Reputation: 1387

Yes—you can set up an export "sink" in Stackdriver to get the data into BigQuery for analysis. They also provide documentation of how the payload fields are transformed on export.

The first link includes some notes on filtering to just the BigQueryAuditMetadata type, but note that you can also filter on fields inside of the payload by using an advanced log filter (e.g. if you only want to see which tables are being written, read, only include certain users, etc.).

Obviously you can just stream everything into BigQuery if you want, but filtering can help keep the noise down.

Upvotes: 2

Related Questions