CODEWITHSUNDEEP
filterlogstash
user3629259
user3629259

Reputation: 21

Parsing out text from a string using a logstash filter

I have an Apache Access Log that I would like to parse out some text from within the REQUEST field:

GET /foo/bar?contentId=ABC&_=1212121212 HTTP/1.1"

What I would like to do is extract and assign the 12121212122 to a value but the value is based off of the prefix ABC&_ (so I think I need an if statement or something). The prefix could take on other forms (e.g., DDD&_)

So basically I would like to say

if (prefix == ABC&_)
   ABCID = 1212121212
elseif (prefix == DDD&_)
   DDDID = <whatever value>
else
   do nothing

I have been struggling to build the right filter in logstash to extract the id based on the prefix. Any help would be great.

Thank you

Upvotes: 0

Views: 1029

Answers (1)

pandaadb
pandaadb

Reputation: 6466

For this you would use a grok filter.

For example: 

artur@pandaadb:~/dev/logstash$ ./logstash-2.3.2/bin/logstash -f conf2
Settings: Default pipeline workers: 8
Pipeline main started
GET /foo/bar?contentId=ABC&_=1212121212 HTTP/1.1"
{
       "message" => "GET /foo/bar?contentId=ABC&_=1212121212 HTTP/1.1\"",
      "@version" => "1",
    "@timestamp" => "2016-07-28T15:59:12.787Z",
          "host" => "pandaadb",
        "prefix" => "ABC&_",
            "id" => "1212121212"
}

This is your sample input, parsing out your prefix and Id.

There is no need for an if here, since the regular expression of the GROK filter takes care of it.

You can however (if you need to put it in different fields) analyse your field and add it to a different one.

This would output like that:

GET /foo/bar?contentId=ABC&_=1212121212 HTTP/1.1"
{
       "message" => "GET /foo/bar?contentId=ABC&_=1212121212 HTTP/1.1\"",
      "@version" => "1",
    "@timestamp" => "2016-07-28T16:05:07.442Z",
          "host" => "pandaadb",
        "prefix" => "ABC&_",
            "id" => "1212121212",
         "ABCID" => "1212121212"
}
GET /foo/bar?contentId=DDD&_=1212121212 HTTP/1.1"
{
       "message" => "GET /foo/bar?contentId=DDD&_=1212121212 HTTP/1.1\"",
      "@version" => "1",
    "@timestamp" => "2016-07-28T16:05:20.026Z",
          "host" => "pandaadb",
        "prefix" => "DDD&_",
            "id" => "1212121212",
         "DDDID" => "1212121212"
}

The filter I used for this looks like that:

filter {
    grok {
        match => {"message" => ".*contentId=%{GREEDYDATA:prefix}=%{NUMBER:id}"}

    }

    if [prefix] =~ "ABC" {
         mutate {
            add_field => {"ABCID" => "%{id}"}
         }
    }

    if [prefix] =~ "DDD" {
         mutate {
            add_field => {"DDDID" => "%{id}"}
         }
    }

}

I hope that illustrates how to go about it. You can use this to test your grok regex:

http://grokdebug.herokuapp.com/

Have fun!

Artur

Upvotes: 2

Related Questions