CODEWITHSUNDEEP
apachelogstashelastic-stack
Alex Zel
Alex Zel

Reputation: 688

Logstash not applying filter to Apache logs

I'me trying to parse some Apache access logs using ELK stack, but I'm having issues with logstash not applying the Apache filter i created on any Apache logs. Here is my filter file:

filter {
  if [type] == "apache_access" {
    grok {
      patterns_dir => ["/opt/logstash/patterns/apache"]
      add_tag => ["grokked", "apache"]
      match => ["messege", "%{IP:client} - - \[%{HTTPDATE:event_date}\] %{QS:first} %{NUMBER:response} %{NUMBER:bytes} %{QS:destination} %{QS:browser}"]
    }
  }
}

filebeat config:

filebeat:
  prospectors:
    -
      paths:
        - /var/log/apache2/access.log
      document_type: apache_access
  registry_file: /var/lib/filebeat/registry

Also I'm using an example log file from logz.io, it contains logs like the following:

88.114.162.149 - - [04/Aug/2016:00:00:05 +0000] "GET /item/giftcards/3802 HTTP/1.1" 200 82 "/category/books" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:9.0.1) Gecko/20100101 Firefox/9.0.1"
156.141.192.36 - - [04/Aug/2016:00:00:10 +0000] "GET /category/toys?from=20 HTTP/1.1" 200 135 "/category/toys" "Mozilla/5.0 (Windows NT 6.0) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.56 Safari/535.11"
92.213.110.215 - - [04/Aug/2016:00:00:15 +0000] "GET /category/software HTTP/1.1" 200 108 "/category/books" "Mozilla/5.0 (Windows NT 6.0) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.56 Safari/535.11"
80.225.119.24 - - [04/Aug/2016:00:00:20 +0000] "GET /category/cameras HTTP/1.1" 200 100 "http://www.google.com/search?ie=UTF-8&q=google&sclient=psy-ab&q=Cameras+Books&oq=Cameras+Books&aq=f&aqi=g-vL1&aql=&pbx=1&bav=on.2,or.r_gc.r_pw.r_qf.,cf.osb&biw=2640&bih=427" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; YTB730; GTB7.2; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; .NET4.0C; .NET4.0E; Media Center PC 6.0)"
208.219.150.176 - - [04/Aug/2016:00:00:25 +0000] "GET /category/software HTTP/1.1" 200 117 "-" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; GTB7.2; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C)"
160.165.186.172 - - [04/Aug/2016:00:00:30 +0000] "GET /category/office HTTP/1.1" 200 101 "/category/electronics" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; YTB720; GTB7.2; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729)"
224.150.219.97 - - [04/Aug/2016:00:00:35 +0000] "GET /category/jewelry HTTP/1.1" 200 74 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

I've check my filter in grokdebug and everything works fine there, but every time I push those logs into logstash it doesn't apply that filter, instead all logs entries have a "_grokparsefailure" tag.

Any idea what could be the issue here? I've followed several guides and still have this problem.

P.S. I know about COMBINEDAPACHELOG but I still wanted to parse it this way for my own experience and to understand ELK stack batter.

Upvotes: 0

Views: 449

Answers (1)

Val
Val

Reputation: 217304

Try to change messege to message in your grok match

         change 'e' to 'a'
                 |
                 v
  match => ["message", "%{IP:client} - - \[%{HTTPDATE:event_date}\] %{QS:first} %{NUMBER:response} %{NUMBER:bytes} %{QS:destination} %{QS:browser}"]

Upvotes: 1

Related Questions