CODEWITHSUNDEEP
elasticsearchlogstashelastic-stackfilebeat
Matthias Lassnig
Matthias Lassnig

Reputation: 91

Elasticsearch Logstash Filebeat mapping

Im having a problem with ELK Stack + Filebeat.

Filebeat is sending apache-like logs to Logstash, which should be parsing the lines. Elasticsearch should be storing the split data in fields so i can visualize them using Kibana.

Problem: Elasticsearch recieves the logs but stores them in a single "message" field.

Desired solution:

Input:

10.0.0.1 some.hostname.at - [27/Jun/2017:23:59:59 +0200]

ES:

"ip":"10.0.0.1"

"hostname":"some.hostname.at"

"timestamp":"27/Jun/2017:23:59:59 +0200"

My logstash configuration:

    input {

  beats {
    port => 5044
  }



}

filter { 


  if [type] == "web-apache" {
        grok {
            patterns_dir => ["./patterns"]
            match => { "message" => "IP: %{IPV4:client_ip}, Hostname: %{HOSTNAME:hostname}, - \[timestamp: %{HTTPDATE:timestamp}\]" }
            break_on_match => false
            remove_field => [ "message" ]
        }

        date {
            locale => "en"
            timezone => "Europe/Vienna"
            match =>  [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
        }

        useragent {
            source => "agent"
            prefix => "browser_"
        }
    }

    }

output {

    stdout {
       codec => rubydebug
    }

    elasticsearch {
             hosts => ["localhost:9200"]
         index => "test1"
             document_type => "accessAPI"
           }
}

My Elasticsearch discover output:

The "message" field contains my whole log file

I hope there are any ELK experts around that can help me. Thank you in advance, Matthias

Upvotes: 1

Views: 965

Answers (1)

PtX
PtX

Reputation: 68

The grok filter you stated will not work here.

Try using:

%{IPV4:client_ip} %{HOSTNAME:hostname} - \[%{HTTPDATE:timestamp}\]


There is no need to specify desired names seperately in front of the field names (you're not trying to format the message here, but to extract seperate fields), just stating the field name in brackets after the ':' will lead to the result you want.

Also, use the overwrite-function instead of remove_field for message. More information here:
https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html#plugins-filters-grok-options

It will look similar to that in the end:

filter {
  grok {
    match => { "message" => "%{IPV4:client_ip} %{HOSTNAME:hostname} - \[%{HTTPDATE:timestamp}\]" }
    overwrite => [ "message" ]
  }
}

You can test grok filters here:
http://grokconstructor.appspot.com/do/match

Upvotes: 1

Related Questions