CODEWITHSUNDEEP
regexlogstashlogstash-grok
user2609980
user2609980

Reputation: 10474

Is it possible to use Logstash to remove_field if it does not match a certain value?

We want to filter a log using Logstash by removing fields if the field does not contain "_log". The remove_field syntax is available, but only works for removing a field if it matches a certain condition. 

filter {
  grok {
    remove_field => [ "log_" ]
  }
}
# This works for removing the log_ field, we want to remove everything that does NOT match log_.

Is it also possible to remove a field if it does not match a certain condition?

We tried using a regex that did just that, but that did not work (is it documented somewhere that you cannot use a regex?). Removing all other fields is also an option, but way more effort. We hope someone can help us fitering all fields that do not contain "log_".

Upvotes: 1

Views: 4788

Answers (2)

Alain Collins
Alain Collins

Reputation: 16362

The regexp should work:

filter {
  if [field] !~ /pattern/ {
    mutate {
      remove_field => [ "field" ]
    }
  }
}

Upvotes: 5

karthik manchala
karthik manchala

Reputation: 13640

One of the ways you can do it is..

  • Tag the fields that do not match the condition (using tag_on_failure)
  • Remove the above tagged fields

Example:-

grok {
    match => ["log_"]
    tag_on_failure => ["_todelete"]
}

and then

grok {
    remove_tag => [ "_todelete" ]
}

Upvotes: 4

Related Questions