CODEWITHSUNDEEP
elasticsearchlogstashkibanafluentdfluent-bit
Naveen
Naveen

Reputation: 145

Fluentbit not sending EKS logs to S3

I've tried to send EKS logs to S3, but logs are not being pushed to S3. But, I can able to visualize logs in Elastic search. Below is the output section of my fb-configmap.conf file.

    output-elasticsearch.conf: |
      [OUTPUT]
          Name            es
          Match           *
          Host            ${FLUENT_ELASTICSEARCH_HOST}
          Port            ${FLUENT_ELASTICSEARCH_PORT}
          HTTP_User       ${FLUENT_ELASTICSEARCH_USER}
          HTTP_Passwd     ${FLUENT_ELASTICSEARCH_PASSWD}
          Logstash_Format Off
          Index           dev
          Replace_Dots    On
          Retry_Limit     False
    output-s3.conf: |
      [OUTPUT]
          Name                  s3
          Match                 *
          bucket                eks-logs
          region                ap-south-1
          total_file_size       250M
          s3_key_format         /%Y/%m/%d/%H/%M/%S/$UUID.gz
          s3_key_format_tag_delimiters .-

As per my understanding, AWS access & secret key will be taken from the .aws/credentials path.
And do I need to install any plugin like fluent-bit-s3-plugin for this?

I've been trying to resolve this for days, any help is much appreciated.

Upvotes: 2

Views: 1860

Answers (1)

Naveen
Naveen

Reputation: 145

Can able to push to S3 after checking the below things,

  • output-s3.conf should be given properly
  • your S3 bucket should have public access
  • should have @INCLUDE output-*.conf in your conf
  • fluent-bit image should be latest/stable/updated

Upvotes: 1

Related Questions