CODEWITHSUNDEEP
logstashlogstash-groklogstash-configuration
geekscrap
geekscrap

Reputation: 1045

logstash add_field and remove_field

I'm attempting to simplify my logstash config. I want to split the program field into separate fields (as show below) however I would prefer to use just one grok statement (if it's at all possible!)

Of the two examples below I get an _grokparsefailure on the second example, but not the first. Since grok has the add_field and remove_field options I would assume that I could combine it all into one grok statement. Why is this not the case? Have I missed some ordering/syntax somewhere?

Sample log:

2016-02-16T16:42:06Z ubuntu docker/THISTESTNAME[892]: 172.16.229.1 - - [16/Feb/2016:16:42:06 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.109 Safari/537.36" "-"

Why does this work:

filter {

       # Extracts the docker name, ID, image etc elements
       mutate {
               add_field => { "[@metadata][program]" => "%{program}" }
               remove_field => "[program]"
       }

       grok {
                patterns_dir => "/logstash/patterns_dir/docker"
                match => { "[@metadata][program]" => "%{D_ID}" }
       }

}

But this does not:

filter {

        grok {
               add_field => { "[@metadata][program]" => "%{program}" }
               remove_field => "[program]"
               patterns_dir => "/logstash/patterns_dir/docker"
               match => { "[@metadata][program]" => "%{D_ID}" }
        }

}

Upvotes: 1

Views: 12880

Answers (2)

geekscrap
geekscrap

Reputation: 1045

This was directly answered by @Alan, however I found this way a little more readable and compressed my code even more:

grok {
    patterns_dir => "/logstash/patterns_dir/docker-patterns"
    match => { "program" => "%{D_ID}" }
    overwrite => [ "program" ]
}

Upvotes: 0

Alain Collins
Alain Collins

Reputation: 16362

add_field and remove_field only run if the underlying filter works. In your second example, the [@metadata][program] doesn't yet exist for you to run grok{} against.

Upvotes: 1

Related Questions