CODEWITHSUNDEEP
google-cloud-platformloggingfilterexploreraudit-logging
Jay A
Jay A

Reputation: 1

GCP log explorer filter for list item count more than 1

I am trying to write a filter in GCP log explorer, which can look for a count of the values of an attribute.

Example: I am trying to find the logs like below, which has two items for "referencedTables" attribute. GCP Log Explorer Screenshot

I have tried below options which doesn't work -

protoPayload.metadata.jobChange.job.jobStats.queryStats.referencedTables.*.count>1

protoPayload.metadata.jobChange.job.jobStats.queryStats.referencedTables.count>1

Also tried Regex looking for "tables" keyword occurrence twice -

protoPayload.metadata.jobChange.job.jobStats.queryStats.referencedTable=~"(\tables+::\tables+))"

Also tried Regex querying second item, which means there are more than one items -

protoPayload.metadata.jobChange.job.jobStats.queryStats.referencedTables1=~"^[A-Za-z0-9_.]+$"

Note that - these types of logs are BigQuery audit logs, that are logged in GCP logging service, when you run "insert into.. select" type of queries in BigQuery.

Upvotes: 0

Views: 1775

Answers (1)

DazWilkin
DazWilkin

Reputation: 40296

I think you can't use logging filters to filter across log entries only within a log entry.

One solution to your problem is log-based metrics where you'd create a metric by extracting values from logs but you'd then have to use MQL to query (e.g. count) the metric.

A more simple (albeit ad hoc) solution is to use use gcloud logging read to --filter the logs (possibly --format the results in JSON for easier processing) and then pipeline the results into a tool like jq where you could count the results.

Upvotes: 1

Related Questions