CODEWITHSUNDEEP
logstashlogstash-configuration
Kelvin
Kelvin

Reputation: 1367

Logstash remove fields by regex

I'm using a FIX filter plugin to process some of our FIX logs. In those messages we receive multiple custom fields. This is outside of the grok filter. I pass the message I care about into this secondary fix plugin

Some of our messages for example look like this:

  "unknown_fields" => [
    [0] "5000",
    [1] "9723",
  ],
  "5000" => "FOOBARVAL",
  "9723" => "BAZBOOHUM",
  "IDSource" => "RIC_CODE",

Question

Is there a way that I can remove tags with mutate or some other filter based on a regular expression(^\d+$)?

More specifically, is there a way that I can remove all of the integer fields that I know will be custom FIX fields (eg. 5000)?

Upvotes: 1

Views: 2580

Answers (2)

Kelvin
Kelvin

Reputation: 1367

I appreciate the other answer, but I ended up using the prune filter plugin.

prune {
    blacklist_names => ["[0-9]+", "unknown_fields", "tags"]
}

Upvotes: 5

Alcanzar
Alcanzar

Reputation: 17155

This answer is very similar to this one: https://stackoverflow.com/a/27396056/2785358

You can do something like this to match a regex:

filter {
  ruby {
    code => "
      event.to_hash.keys.each { |k|
        if k.match(/^\d+$/
          event.remove(k)
        end
      }"
  }
}

Upvotes: 1

Related Questions