CODEWITHSUNDEEP
jakarta-eeloggingmonitoringlogstash
Martin V.
Martin V.

Reputation: 3710

Logstash multiline filter configuration and java exceptions

We started recently to gather all java server logs with logstash.

Default log4j configuration works fine, but for exceptions and sql queries we use :

http://logstash.net/docs/1.2.2/filters/multiline

filter {
 multiline {
  type => "somefiletype"
  pattern => "^\s"
  what => "previous"
 }
}

but this does not work 100% precisely, some Exceptions/SQLs does not have desired format.

Would be possible to configure multiline plugin in the way that :

IF there's not timestamp on the beggining of the line it's multiline log message ?

Upvotes: 2

Views: 10285

Answers (2)

Farrukh Najmi
Farrukh Najmi

Reputation: 5316

I finally had success with parsing multi-line log messages from my log files. Note that your log messages may be slightly different requiring change in the grok pattern. The main fix was to place multiline filter before grok filter and do the drop{} between it and the grok filter. Important: Use the Grok Debugger to debug your grok filters.

input {
  stdin {
    type => "log4j"
  }
}

filter {
  if [type] == "log4j" {

    multiline {
      pattern => "^[\d]{4}\-[\d]{2}\-[\d]{2} "
      negate => true
      what => previous
    }
    if "_grokparsefailure" in [tags] {
      drop { }
    } 
    grok {
      match => {
      "message" =>  "(?<logdate>[\d]{4}\-[\d]{2}\-[\d]{2} [\d]{2}:[\d]{2}:[\d]{2},[\d]{3})%{SPACE}%{NUMBER:unknown1}%{SPACE}%{LOGLEVEL:severity}%{SPACE}\[(?<logger>[^\]]+)\]%{SPACE}\((?<thread>[^\)]+)\)%{SPACE}%{GREEDYDATA:message}"
      }
      overwrite => [ "message" ]
    }

    if !("_grokparsefailure" in [tags]) {
      date {
        match => [ "logdate", "YYYY-MM-dd HH:mm:ss,SSS"]
      }
    }
  }
}


output {   
  # Print each event to stdout.
  stdout {
    codec => json
  }
}

Upvotes: 1

Elvar
Elvar

Reputation: 454

Use the multiline codec instead, the docs even have an example for this exact problem http://logstash.net/docs/1.2.2/codecs/multiline

Upvotes: 5

Related Questions