CODEWITHSUNDEEP
logstashlogstash-grokelastic-stackfilebeat
rayhan
rayhan

Reputation: 646

Logstash: Unable to filter apache vhost_combined custom log and post them into Elasticsearch

I need to filter following apache vhost log format with Geo IP. So that grok can understand and work smoothly.

Exact log format: 

LogFormat "%V:%p %h %l %u %t \"%r\" %>s %O %T %D \"%{Referer}i\" \"%{User-Agent}i\"" vhost_combined_custom

You see %T and %D flag are extra here.

Message:

sub1.example.com:443 1.9.202.41 - - [03/Jun/2016:06:58:17 +0000] "GET /notifications/pendingCount HTTP/1.1" 200 591 0 32165 "https://sub1.example.com/path/index?var=871190" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36"

Any suggestions?

Upvotes: 0

Views: 637

Answers (2)

tread
tread

Reputation: 11098

You can also do:

match => { "message" => "%{HOSTNAME:vhost}\:%{NUMBER:port} %{COMBINEDAPACHELOG}"}

As per Rene's blog

Upvotes: 1

rayhan
rayhan

Reputation: 646

Ok, I found the solution after multiple testing. It should be like this:

grok {                                                                                           
      match => { "message" => "%{IPORHOST:vhost}:%{POSINT:port} %{IPORHOST:clientip} %{USER:ident} %{USER:auth} \[%{HTTPDATE:timestamp}\] \"(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})\" %{NUMBER:response} %{NUMBER:bytes} %{NUMBER:seconds} %{NUMBER:microseconds} %{QS:referrer} %{QS:agent}" }
}

Upvotes: 0

Related Questions