CODEWITHSUNDEEP
logstashxpackelastic-cloud
Mauricio Rodrigues
Mauricio Rodrigues

Reputation: 807

Receiving "Empty configuration for pipeline_id" error when try send data from logstash to Elastic Cloud

Additional info:

Problem:

Hi,

I'm trying send data from logstash to Elastic Cloud, but i'm receiving the follow error when logstash run:

"Empty configuration for pipeline_id: artist_profile_views"

But if i try run logstash without define xpack configurations in logststah.yml, this work fine and the stdout {} send collected data to output.

Please, see my configurations files:

logststash.yml

cloud.id: "[...]"
cloud.auth: "[user]:[password]"
xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.url: https://elasticCloudUrl
xpack.monitoring.elasticsearch.username: [user]
xpack.monitoring.elasticsearch.password: [password]
xpack.management.enabled: true
xpack.management.pipeline.id: ["artist_profile_views", "searched"]
xpack.management.elasticsearch.username: [user]
xpack.management.elasticsearch.password: [password]
xpack.management.elasticsearch.url: ["https://elasticCloudUrl"]

pipelines.yml

- pipeline.id: artist_profile_views
  path.config: "pipelines/artist_profile_views.conf"

- pipeline.id: searched
  path.config: "pipelines/searched.conf"

artist_profile_views

input { 
    file {
        path => "/Users/zabaala/Sites/cna/stats/artist_profile_views/artist_profile_views_*.log"
        codec => json
        start_position => "beginning"
    }
}

filter {
    geoip {
        source => "[ip]"
    }   
    useragent {
        source => "[headers][user_agent]"
        target => "[headers][request]"
    }
    mutate {
        remove_field => ["[headers][user_agent]"]
    }
}


output {
    elasticsearch { 
        # hosts => ["https://ElasticCloudUrl"] 
        index => "stats"
    }
    stdout { 
        codec => rubydebug
    }
}

Sample data:

{"artist_profile_views":{"id":"510","type":"ARTIST","area":"PHOTOS"},"env":"local","ip":"172.18.0.1","index":"stats","doc":"artist_profile_views","when":{"date":"2018-07-06T17:20:48-0300"},"viewer":{"id":null,"context":"GUEST","by_himself":true},"headers":{"user_agent":"Mozilla\/5.0 (Macintosh; Intel Mac OS X 10_13_4) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/67.0.3396.99 Safari\/537.36"}}
{"artist_profile_views":{"id":"510","type":"ARTIST","area":"EVENTS"},"env":"local","ip":"172.18.0.1","index":"stats","doc":"artist_profile_views","when":{"date":"2018-07-06T17:20:50-0300"},"viewer":{"id":null,"context":"GUEST","by_himself":true},"headers":{"user_agent":"Mozilla\/5.0 (Macintosh; Intel Mac OS X 10_13_4) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/67.0.3396.99 Safari\/537.36"}}
{"artist_profile_views":{"id":"510","type":"ARTIST","area":"AREA_AUDIOS"},"env":"local","ip":"172.18.0.1","index":"stats","doc":"artist_profile_views","when":{"date":"2018-07-06T17:20:52-0300"},"viewer":{"id":null,"context":"GUEST","by_himself":true},"headers":{"user_agent":"Mozilla\/5.0 (Macintosh; Intel Mac OS X 10_13_4) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/67.0.3396.99 Safari\/537.36"}}
{"artist_profile_views":{"id":"510","type":"ARTIST","area":"VIDEOS"},"env":"local","ip":"172.18.0.1","index":"stats","doc":"artist_profile_views","when":{"date":"2018-07-06T17:20:55-0300"},"viewer":{"id":null,"context":"GUEST","by_himself":true},"headers":{"user_agent":"Mozilla\/5.0 (Macintosh; Intel Mac OS X 10_13_4) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/67.0.3396.99 Safari\/537.36"}}
{"artist_profile_views":{"id":"510","type":"ARTIST","area":"HOME"},"env":"local","ip":"172.18.0.1","index":"stats","doc":"artist_profile_views","when":{"date":"2018-07-06T17:31:32-0300"},"viewer":{"id":null,"context":"GUEST","by_himself":true},"headers":{"user_agent":"Mozilla\/5.0 (Macintosh; Intel Mac OS X 10_13_4) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/67.0.3396.99 Safari\/537.36"}}
{"artist_profile_views":{"id":"510","type":"ARTIST","area":"AREA_AUDIOS"},"env":"local","ip":"172.18.0.1","index":"stats","doc":"artist_profile_views","when":{"date":"2018-07-06T17:31:43-0300"},"viewer":{"id":null,"context":"GUEST","by_himself":true},"headers":{"user_agent":"Mozilla\/5.0 (Macintosh; Intel Mac OS X 10_13_4) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/67.0.3396.99 Safari\/537.36"}}

What's wrong?

Thanks.

Upvotes: 0

Views: 1104

Answers (1)

Mauricio Rodrigues
Mauricio Rodrigues

Reputation: 807

I found the answer.

This problem occur because the configuration xpack.managed.* use the Centralized Pipeline feature.

With centralized pipeline you don't need create any pipeline.yml file to configure a local logstash pipeline. With centralized pipeline, you put you own pipelines into elasticsearch. This is a resource of the xpack.

When you configure your logstash to be managed by elasticsearch, the pipelines will be loaded from logstash by id.

More about centralized pipeline can be found here: https://www.elastic.co/guide/en/logstash/current/logstash-centralized-pipeline-management.html

Upvotes: 2

Related Questions