CODEWITHSUNDEEP
elasticsearchamazon-ecsamazon-cloudwatchfluent-bit
stein korsveien
stein korsveien

Reputation: 1117

How do I connect AWS ECS to ElasticSearch with Fluentbit?

I want to connect my AWS ECS with Elasticsearch. I have written a Pulumi script where I > shuold use the fluentbit docker component as a sidecar to my frontend and backend application components frontend and backend. The Pulumi script looks like this:

import * as awsx from "@pulumi/awsx";
export const backendLb = new awsx.lb.ApplicationListener("backend-loadbalance", { port: 80 });
export const frontendLb = new awsx.lb.ApplicationListener("frontend-loadbalance", { port: 80, protocol:"HTTP" });
 export const service = new awsx.ecs.FargateService("backend", {
    taskDefinitionArgs: {   
                          containers: {  
                                         logRuter: {          
                                         
                                           image: "docker.io/amazon/aws-for-fluent-bit:latest",
                                           essential: true,
                                           firelensConfiguration: {
                                                                    "type": "fluentbit",
                                                                    "options":{
                                                                                 "enable-ecs-log-metadata":"true"
                                                                               }
                                                                   },
                                           logConfiguration: {
                                                                logDriver: "awslogs",
                                                                options: {
                                                                            "awslogs-create-group": "true",
                                                                            "awslogs-group": "fluent-bit-cloudwatch",
                                                                            "awslogs-region": "eu-north-1",
                                                                            "awslogs-stream-prefix": "ecs"
                                                                         }
                                                              },
                                            memoryReservation: 50
                                            },
                                           backend: { 
                                                       image: 'steinko/helloworld-backend',
                                                       portMappings: [ backendLb ],
                                                       logConfiguration: {
                                                                            logDriver: "awsfirelens",
                                                                            options: { 
                                                                                         "Name":"es",
                                                                                         "Port": "9243",
                                                                                         "Cloud_ID": "xxxxxxxxxx",
                                                                                         "Cloud_Auth":"elastic:xxxxxxxxxx",
                                                                                         "match":"*",
                                                                                         "Index": "backend",
                                                                                         "Type":"doc",
                                                                                         "tls": "On",
                                                                                         "tls.verify": "Off"
                                                                                         
                                                                                     }..............

The CloudWatch log for the

2

022-03-04T16:30:14.971+01:00    [1mFluent Bit v1.8.12[0m

2022-03-04T16:30:14.971+01:00   * [1m[93mCopyright (C) 2019-2021 The Fluent Bit Authors[0m

2022-03-04T16:30:14.971+01:00   * [1m[93mCopyright (C) 2015-2018 Treasure Data[0m

2022-03-04T16:30:14.971+01:00   * Fluent Bit is a CNCF sub-project under the umbrella of Fluentd

2022-03-04T16:30:14.971+01:00   * https://fluentbit.io

2022-03-04T16:30:14.982+01:00   [2022/03/04 15:30:14] [ info] [engine] started (pid=1)

2022-03-04T16:30:15.063+01:00   [2022/03/04 15:30:14] [ info] [storage] version=1.1.5, initializing...

2022-03-04T16:30:15.063+01:00   [2022/03/04 15:30:14] [ info] [storage] in-memory

2022-03-04T16:30:15.063+01:00   [2022/03/04 15:30:14] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128

2022-03-04T16:30:15.063+01:00   [2022/03/04 15:30:14] [ info] [cmetrics] version=0.2.2

2022-03-04T16:30:15.063+01:00   [2022/03/04 15:30:14] [ info] [input:tcp:tcp.0] listening on 127.0.0.1:8877

2022-03-04T16:30:15.063+01:00   [2022/03/04 15:30:14] [ info] [input:forward:forward.1] listening on unix:///var/run/fluent.sock

2022-03-04T16:30:15.063+01:00   [2022/03/04 15:30:14] [ info] [input:forward:forward.2] listening on 127.0.0.1:24224

2022-03-04T16:30:15.164+01:00   [2022/03/04 15:30:15] [ info] [sp] stream processor started

2022-03-04T16:30:19.942+01:00   [2022/03/04 15:30:19] [error] [output:es:es.1] HTTP status=400 URI=/_bulk, response:

2022-03-04T16:30:19.942+01:00   {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Action/metadata line [1] contains an unknown parameter [_type]"}],"type":"illegal_argument_exception","reason":"Action/metadata line [1] contains an unknown parameter [_type]"},"status":400}

2022-03-04T16:30:19.942+01:00   [2022/03/04 15:30:19] [ warn] [engine] failed to flush chunk '1-1646407816.588524885.flb', retry in 7 seconds: task_id=0, input=forward.1 > output=es.1 (out_id=1)

2022-03-04T16:30:26.486+01:00   [2022/03/04 15:30:26] [error] [output:es:es.1] HTTP status=400 URI=/_bulk, response:

2022-03-04T16:30:26.486+01:00   {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Action/metadata line [1] contains an unknown parameter [_type]"}],"type":"illegal_argument_exception","reason":"Action/metadata line [1] contains an unknown parameter [_type]"},"status":400}

2022-03-04T16:30:26.486+01:00   [2022/03/04 15:30:26] [ warn] [engine] chunk '1-1646407816.588524885.flb' cannot be retried:

The detail error message look like this:

[2022/03/04 13:04:33] [error] [output:es:es.1] HTTP status=400 URI=/_bulk, response:


{
    "error": {
        "root_cause": [
            {
                "type": "illegal_argument_exception",
                "reason": "Action/metadata line [1] contains an unknown parameter [_type]"
            }
        ],
        "type": "illegal_argument_exception",
        "reason": "Action/metadata line [1] contains an unknown parameter [_type]"
    },
    "status": 400
}

How do I fix this error

Upvotes: 1

Views: 1794

Answers (1)

venlentine
venlentine

Reputation: 53

update you fluent.conf set

[OUTPUT]
    Name             es
    Match            *
    Host             efk-elastic
    Port             9200
    Logstash_format  on
    Replace_dots     on
    Retry_limit      false
    Type             doc
    Suppress_Type_Name On

Upvotes: 4

Related Questions