CODEWITHSUNDEEP
elasticsearchgoogle-cloud-platformstackdriver
Dark Knight
Dark Knight

Reputation: 8347

List error logs from Stckdriver of matching pattern

I am evaluating approaches for a scenario where i need to fetch list of logs from Stackdriver. There can be multiple Filter criteria's (for eg. payload contains a word 'retry' of logs of type 'Warning' ...)

With help gcp sdk i was able to query stackdriver but not sure how efficient is this approach. Kindly suggest other approaches where i can use elastic search client to query stackdriver and list matching logs

Upvotes: 3

Views: 164

Answers (1)

Himal
Himal

Reputation: 3482

It looks like you have multiple sets of logs that you wish to consume separately and each of those log sets can be described with a Stackdriver filter. This is a good start since running filters against Stackdriver is an effective way to sort your data. And you are right that running the same filter against Stackdriver over and over again would be pretty inefficient.

The following approach uses Stackdriver log sinks and this is how we manage logs on our GCP account. Our monitoring team is pretty happy with it and it's easy to maintain.

You can read up on log sinks here and aggregated log sinks here.

The general idea is to have Google automatically filter and export the logs for you using multiple log sinks (one sink per filter). The export destination can be Google Storage, BigQuery, or Pub/Sub. Each sink should export to a different location and will do so continuously as long as the sink exists. Also, log sinks can be set up per project or at the organization level (where it can inherit all projects underneath).

For example, let's say you want to set up three log sinks. Each sink uses a different filter and different export location (but all to the same bucket):

  • Log Sink 1 (compute logs) -> gs://my-example-log-dump-bucket/compute-logs/
  • Log Sink 2 (network logs) -> gs://my-example-log-dump-bucket/network-logs/
  • Log Sink 3 (linux logs) -> gs://my-example-log-dump-bucket/linux-logs/

Once this is set up, your code's SDK can just access each location based on what logs it currently needs. This eliminates the need for your code to do the filtering since Google has already handled it for you in the background.

One thing to note: log exports to BigQuery and Pub/Sub are instant, but exports to Google Storage occur at the top of every hour. So if you need a fast turnaround on the logs, avoid Google Storage and go with either BigQuery or Pub/Sub.

Hope this helps!

Upvotes: 3

Related Questions