CODEWITHSUNDEEP
elasticsearchlogstashkibanaelastic-stackfilebeat
Don Draper
Don Draper

Reputation: 33

Create pipeline for filebeat

I've enabled the filebeat system module:

filebeat modules enable system
filebeat setup --pipelines --modules system
filebeat setup --dashboards
systemctl restart filebeat

This is what logstash has to say pipeline with id [filebeat-7.9.0-system-auth-pipeline] does not exist

This is the part of logstash that is responsible for it:

output {
    if [@metadata][pipeline] {
        elasticsearch {
            hosts => "https://localhost:9200"
            manage_template => false
            cacert => "/etc/elasticsearch/estackcap12extract.crt"
            ssl => true
            ssl_certificate_verification => false
            index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
            pipeline => "%{[@metadata][pipeline]}"
            user => "elastic"
            password => "*secret*"
        }
    } else {
    ...

Do I need to manually create a pipeline for this? Did I do something wrong? Best I could find is this doc page page but it seem to be for custom things and this is a ready module so I'm not sure how relevant it is.

Upvotes: 1

Views: 2139

Answers (1)

Val
Val

Reputation: 217564

The issue is that your Filebeat doesn't connect to ES directly but only through Logstash. It's a known issue, but since *Beat can only have as single output, you need to do the following trick.

What you need to do is to uncomment the elasticsearch output just when running the setup command, so that Filebeat can install ingest pipelines.

When done, you need to comment out that output again and uncomment the Logstash one before starting Filebeat for real.

If you don't want to modify your configuration file, there's another way by passing configuration variables to filebeat setup, like this:

filebeat setup --pipelines --modules system \
    -E output.logstash.enabled=false \
    -E output.elasticsearch.username="elastic" \
    -E output.elasticsearch.password="*secret*" \
    -E 'output.elasticsearch.ssl.certificate_authorities="/etc/elasticsearch/estackcap12extract.crt"' \
    -E output.elasticsearch.hosts="https://localhost:9200"

Upvotes: 1

Related Questions