CODEWITHSUNDEEP
nginxgoogle-cloud-platformfluentdstackdriver
James M
James M

Reputation: 558

Parsing error messges from nginx log to stackdriver with google-fluend

We trying to set up alerting by parsing nginx error, and we need only messages with [error] level. ([error] 27720#27720: *29765923 access forbidden by rule, client: blalbablab). Main goal is to not send in gcp stackdriver logs that we don't need (decrease logs storage space using). So i created google-fluend with logs streaming, but i'm stuck with parsing error messages. Here is my config.

<source>
  @type tail
  tag nginx-error
  path /var/log/nginx/error.log
  format multiline
  format_firstline /^\d{4}/\d{2}/\d{2} \d{2}:\d{2}:\d{2} \[\w+\] (?<pid>\d+).(?<tid>\d+): /
  format1 /^(?<time>\d{4}/\d{2}/\d{2} \d{2}:\d{2}:\d{2}) \[(?<log_level>\w+)\] (?<pid>\d+).(?<tid>\d+): (?<message>$      multiline_flush_interval 3s
  #<parse>
    #@type regexp
    #expression (\s\[error+\])
    #time_key name
    #types id:integer
  #</parse>
</source>

Is this correct to using parse with regexp rules in this case? Please help me with this.

Upvotes: 1

Views: 503

Answers (1)

James M
James M

Reputation: 558

After some testing, I got a working version.

 <source>
  @type tail
  format /^(?<message>.*\[error\].*)$/
  path /var/log/nginx/error.log
  pos_file /var/lib/google-fluentd/pos/nginx-error.pos
  read_from_head true
  tag nginx-error
</source>

Upvotes: 1

Related Questions