CODEWITHSUNDEEP
javalogstashjrubyelastic-stack
Mitter Singh Thakur
Mitter Singh Thakur

Reputation: 11

logstash 2.3.3 multiline filter not working with multiple workers

I have started logstash with multiple workers > 16.

I have multiline messages like java exceptions/java traces and want to merge them into a single event. Earlier, It was working as expected but after upgrading my ELK stack it's breaking :-(

my logstash filter : 

filter {
  multiline {

    pattern => "(^[a-zA-Z.]+(?:Error|Exception): .+)|(^\s+at .+)|(^\s+... \d+ more)|(^\s*Caused by:.+)"
    what => "previous"
  }
}

logstash logs : 

:message=>"Warning: Manual override - there are filters that might not work with multiple worker threads", :worker_threads=>16, :filters=>["multiline"], :level=>:warn}
Exception in pipelineworker, the pipeline stopped processing new events, please check your filter configuration and restart Logstash.

Upvotes: 1

Views: 805

Answers (1)

baudsp
baudsp

Reputation: 4110

If you have upgraded from Logstash version 1.5, it is expected.
In the version 2.0 was introduced the worker threads, but since the multiline filter is not thread-safe, it prevents using more than one worker thread.

So you'll have to either :

  • Use the multiline codec.
    For example to join java exception stacktrace :

    input {
  stdin {
    codec => multiline {
      pattern => "(^.+Exception: .+)|(^\s+at .+)|(^\s+... \d+ more)|(^\s*Caused by:.+)"
      what => "previous"
    }
  }
}

  • Do the multiline operation on your shipper (both beaver and filebeat can be configured to do it)

  • Use only one worker thread (in that case you can use multiple instances of Logstash to use all your CPU cores, but keep in mind that the multiline filter will be eventually removed).

Upvotes: 2

Related Questions