CODEWITHSUNDEEP
logstashelastic-stacklogstash-configurationfilebeat
James
James

Reputation: 16349

Setting document_type of log in filebeat stops filebeat restarting

I am trying to import custom log I have on my server through filebeat and send it over to logstash for use in my ELK stack.

I have set this up to work correctly and it runs fine currently.

However, I am wishing to add a logstash filter for this specific log and so decided to add a document_type field for this log to allow me to filter based on it in logstash.

I have done this like so:

filebeat.prospectors:
- input_type: log
  paths:
    - /var/log/apache2/access.log
  document_type: apache-access

- input_type: log
  paths:
    - /var/www/webapp/storage/logs/laravel.log

- input_type: log
  paths:
    - /opt/myservice/server/server.log
    document_type: myservice

I have added document_type: myservice to the log for myservice, and believe I have done so according to the documentation here. Furthermore it is done the same as I have done it for the apache access log.

However when I restart filebeat, it won't start back up again. I have tried looking at the log for filebeat - however there doesn't seem to be anything in there about why it won't start.

If I comment out document_type: myservice, like this #document_type: myservice and then restart filebeat it boots up correctly which means it must be something to do with that line?

Questions:

Am I doing something wrong here?

Is there an alternative method I could use to apply my logstash filter to this log only other than using if [type] == "myservice"?

Upvotes: 0

Views: 1032

Answers (1)

A J
A J

Reputation: 2593

Using document_type is a good approach to applying conditionals in Logstash. An alternative method is to apply tags or fields in Filebeat.

The problem with your configuration is the indentation of the document_type: myservice that you added. Notice how the indentation is different than the document_type: apache-access. The document_type field should be at the same level as paths and input_type as they are all prospector options.

You can test your config file with filebeat.sh -c /etc/filebeat/filebeat.yml -e -configtest.

You can also run your config through a tool like http://www.yamllint.com just to check that it's valid YAML.

Upvotes: 1

Related Questions