CODEWITHSUNDEEP
elasticsearchlogstashelastic-stackfilebeat
red888
red888

Reputation: 31662

logstash not reading logtype field from beats

I have logstash filebeat and elasticsearch running on one node.

I'm trying to get logstash to identify logs labeled as "syslog" and dump them in an index named "syslog", but it appears to not see the label as they are all going into the "uncategorized" index (my catch all default index)

Here is my beats config

/etc/filebeat/filebeat.yml
filebeat:
  prospectors:
    -
      paths:
        - /var/log/messages
      fields:
        type: syslog
output:
  logstash:
    hosts: ["localhost:9901"]

Here is my logstash config file

/etc/logstash/conf.d/logstash_server_syslog.conf
input {
    beats {
        port => "9901"
    }
}

filter {
  if [type] == "syslog" {
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
      add_field => [ "received_at", "%{@timestamp}" ]
      add_field => [ "received_from", "%{host}" ]
    }
    date {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
    }
  }
}

output {
  if [type] == "syslog" {
    elasticsearch {
      hosts => ["10.0.0.167:9200", "10.0.0.168:9200"]
      index => "syslog"
    }
  } else {
    elasticsearch {
      hosts => ["10.0.0.167:9200", "10.0.0.168:9200"]
      index => "uncategorized"
    }
  }
}

Upvotes: 0

Views: 564

Answers (2)

A J
A J

Reputation: 2593

To set a custom type field in Filebeat using the document_type configuration option.

filebeat:
  prospectors:
    - paths:
        - /var/log/messages
      document_type: syslog

This will set the @metadata.type field for use with Logstash whereas a custom field will not.

Upvotes: 1

Alain Collins
Alain Collins

Reputation: 16362

Looking at the output (with a stdout{} stanza) would confirm this, but I'm guessing that you missed this part of the doc:

By default, the fields that you specify [in the 'fields' config'] will be grouped under a fields sub-dictionary in the output document. To store the custom fields as top-level fields, set the fields_under_root option to true.

Upvotes: 1

Related Questions