CODEWITHSUNDEEP
logstashlogstash-configuration
Carla
Carla

Reputation: 3380

Logstash: unable to filter lines from metrics


I'd need to collect metrics from an URL. The format of the metrics is like that:

# HELP base:classloader_total_loaded_class_count Displays the total number of classes that have been loaded since the Java virtual machine has started execution.
# TYPE base:classloader_total_loaded_class_count counter
base:classloader_total_loaded_class_count 23003.0

I'd need to exclude, from the events collected, all lines which begin with a '#' character. So I have arranged for the following configuration file:

input {


  http_poller {
    urls => {
      pool_metrics => {
        method => "get"
        url => "http://localhost:10090/metrics"
        headers => {
          "Content-Type" => "text/plain"
        }
      }

}
request_timeout => 30
schedule => { cron => "* * * * * UTC"}
codec => multiline  {
pattern => "^#"
negate => "true"
what => previous
}
type => "server_metrics"
  }
}


output {
  elasticsearch {

    # An index is created for each type of metrics inpout
    index => "logstash-%{type}" 
  }

}

Unfortunately, when I check through elastic search the data collected, I see it's not really what I was expecting. For example:

 {
        "_index" : "logstash-server_metrics",
        "_type" : "doc",
        "_id" : "2egAvWcBwbQ9kTetvX2o",
        "_score" : 1.0,
        "_source" : {
          "type" : "server_metrics",
          "tags" : [
            "multiline"
          ],
          "message" : "# TYPE base:gc_ps_scavenge_count counter\nbase:gc_ps_scavenge_count 24.0",
          "@version" : "1",
          "@timestamp" : "2018-12-17T16:30:01.009Z"
        }
      },

So it seems that the lines with '#' aren't skipped but appended to the next line from the metrics. Can you recommend any way to fix it?

Upvotes: 2

Views: 25

Answers (1)

baudsp
baudsp

Reputation: 4100

The multiline codec doesn't work this way. It merges the events into a single event, appending the lines that don't match ^# as you have observed.

I don't think it's possible to drop messages with a codec, you'll have to use the drop filter instead.

First remove the codec from your input configuration, then add this filter part to your configuration:

filter {
  if [message] =~ "^#" {
    drop {}
  }
}

Using conditionals, if the message matches ^#, the event will be dropped by the drop filter, as you wanted.

Upvotes: 1

Related Questions