CODEWITHSUNDEEP
logstashlogstash-groklogstash-configuration
user3138929
user3138929

Reputation: 389

Match whole file

I'm trying to use logstash to feed a whole text file as a message into ElasticSearch. I'm trying to use the multiline codec but I can't figure out what pattern I have to use. ((.|\n)*) this regex matches all text but this doesn't work for logstash. 

input {
  file {
      path => "/opt/rp/*.txt"
      type => "rp"
      start_position => "beginning"
      stat_interval => 1
      codec => multiline {
          pattern => "((.|\n)*)"
          negate => "false"
          what => "next"
        }
  }

}

What pattern should I use to match all contents of a text file?

Upvotes: 0

Views: 1357

Answers (1)

yeehaw
yeehaw

Reputation: 11

So for me the key to resolve this was adjusting both the input and the filter.

This goes into your input {} :

file {
   path => [ "/opt/rp/*.txt" ]
   start_position => "beginning"
   type => "rp"
}

And this goes into your filter {} :

if [type] == "rp" {
    multiline {
      pattern => "/.*./gm"
      negate => true
      what => "previous"
      add_field => [ "executed_at", "%{@timestamp}" ]
  }
}

So your conf. file needs to have these segments to make it work.

Explanation (afaik):

You use the start_position in the input to get the file read from the beginning to the end instead of the end from beginning (so that the stream reader considers it more of a complete entity instead of an active stream).

Then, you use the multiline filter (not the codec, I tried - doesn't work for this), because it was intended for this purpose. The pattern is the key, that is a RegEx that basically matches everything in your file. Also negate is set to true, so even if it doesn't, negate's gonna treat it as a complete file. You use previous so that the content is added to the previous and not the next entry. The add_field is optional, but I found it useful so that I can establish a time even without the log having a timed name / correct timestamp on it.

This is what I found to be working for me, try it and let us know here if it works (hopefully it does).

Upvotes: 1

Related Questions