CODEWITHSUNDEEP
amazon-web-serviceselasticsearchfilebeat
zangw
zangw

Reputation: 48356

Filebeat failed to connect to Amazon Elasticsearch Service

We try to connect to AWS ES through file beat, However, the following errors comes up

pipeline/output.go:100  Failed to connect to backoff(elasticsearch(https://xxx.us-east-1.es.amazonaws.com:443)): 
Connection marked as failed because the onConnect callback failed: 
cannot retrieve the elasticsearch license: unauthorized access, 
could not connect to the xpack endpoint, verify your credentials

and the config file as

  filebeat.yml: |-
    filebeat.config:
      inputs:
        enabled: true
        path: ${path.config}/inputs.d/*.yml
        reload.enabled: true
        reload.period: 60s
      modules:
        enabled: true
        path: ${path.config}/modules.d/*.yml
        reload.enabled: true
        reload.period: 60s

    output.elasticsearch:
      protocol: "https"
      hosts: ["xxxx.us-east-1.es.amazonaws.com:443"]
      username: "user_id"
      password: "password"
      index: "test-log--%{+yyyy.MM.dd}"
    setup.template.name: "test-log"
    setup.template.pattern: "test-log-*"
    setup.ilm.enabled: false

The filebeat docker image is docker.elastic.co/beats/filebeat:7.13.1

The ES version of Amazon Elasticsearch Service is 7.10

Could someone help us to find the invalid configuration here?

Thanks

Upvotes: 1

Views: 2168

Answers (1)

zangw
zangw

Reputation: 48356

Two issues are here

  1. For Beats 7.13

Beats may not be sending data to some distributions of Elasticsearchedit

we may use the same version filebeat-oss:7.10.2 as aws es 7.10.2

  1. X-Pack settings that need to be disabled.
setup.ilm.enabled: false
setup.pack.security.enabled: false
setup.xpack.graph.enabled: false
setup.xpack.watcher.enabled: false
setup.xpack.monitoring.enabled: false
setup.xpack.reporting.enabled: false

Ref: https://aws.amazon.com/premiumsupport/knowledge-center/es-connect-filebeat-logstash-linux/

Upvotes: 1

Related Questions