CODEWITHSUNDEEP
logstashelastic-stacklogstash-configurationlogstash-forwarder
PiotrL
PiotrL

Reputation: 307

logstash conditional statement in input block

I'm using logstash and lumberjack to manage my logs. Logstash gets input from two different file types, one of which logs multiline stack traces. I know I should use multiline codec, but I don't know how to apply it to only one type of logs. This does not work:

input {
  lumberjack {
    #some configs
    port => 9000

    if [type] == "TYPE1" { # IF STATEMENT CAUSES ERROR
      codec => multiline {}
    }
  }
}

Is there any way to allow input codecs to work with conditionals?

I know I can use multiline filter instead of codec, but multiline filters disallow mutiple workers and I need them.

Upvotes: 2

Views: 1957

Answers (1)

Alex
Alex

Reputation: 23

I suspect the answer is to set up separate inputs for different data types. That way you don't need to add any logic into multiline codec. For example: Send all Java logs that follow pattern X to port N Send all Ruby logs that follow pattern Y to port N+1

I'm not sure how to make this work in a stack that includes Redis or other caching server since all data should ideally go into the cache without any processing, but will require processing logic when it is retrieved since it will contain various data types.

I guess you need to take care of joining the multiline logs at the receiving layer so that caching servers receive complete messages and then run the heavier filters before you send the data to Elastic Search.

Something like this:

LS1 (multiple inputs and codecs like multiline)
Redis (caching)
LS2 (filters)
ES (storage)
KIB (viewing)

Upvotes: 1

Related Questions